{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Using pre-trained embeddings with TensorFlow Hub</h1>\n",
    "\n",
    "This notebook illustrates:\n",
    "<ol>\n",
    "    <li>How to instantiate a TensorFlow Hub module</li>\n",
    "    <li>How to find pre-trained TensorFlow Hub modules for a variety of purposes</li>\n",
    "    <li>How to examine the embeddings of a Hub module</li>\n",
    "    <li>How one Hub module composes representations of sentences from individual words</li>\n",
    "    <li>How to assess word embeddings using a semantic similarity test</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# change these to try this notebook out# chang \n",
    "BUCKET = 'cloud-training-demos-ml'\n",
    "PROJECT = 'cloud-training-demos'\n",
    "REGION = 'us-central1'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the TensorFlow Hub library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q tensorflow-hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import pandas as pd\n",
    "import re\n",
    "import seaborn as sns\n",
    "import scipy\n",
    "import math\n",
    "\n",
    "os.environ['BUCKET'] = BUCKET\n",
    "os.environ['PROJECT'] = PROJECT\n",
    "os.environ['REGION'] = REGION\n",
    "os.environ['TFVERSION'] = '1.8'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py3env/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.8.0\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>TensorFlow Hub Concepts</h2>\n",
    "\n",
    "TensorFlow Hub is a library for the publication, discovery, and consumption of reusable parts of machine learning models. A module is a self-contained piece of a TensorFlow graph, along with its weights and assets, that can be reused across different tasks in a process known as transfer learning, which we covered as part of the course on Image Models.\n",
    "\n",
    "To download and use a module, it's as easy as:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "with tf.Graph().as_default():\n",
    "  module_url = \"path/to/hub/module\"\n",
    "  embed = hub.Module(module_url)\n",
    "  embeddings = embed([\"word1\", \"word2\", \"word3\"])\n",
    "  # ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, because modules are self-contained parts of a TensorFlow graph, in order to actually collect values from a module, you'll need to evaluate it in the context of a session."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "  # .... earlier code\n",
    "  with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    sess.run(tf.tables_initializer())\n",
    "    print(sess.run(embeddings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's explore what hub modules there are. Go to https://www.tensorflow.org/hub/modules/ and explore a bit.\n",
    "\n",
    "Note that TensorFlow Hub has modules for Images, Text, and Other. In this case, we're interested in a Text module, so navigate to the Text section.\n",
    "\n",
    "Within the Text section, there are a number of modules. If you click on a link, you'll be taken to a page that describes the module and links to the original paper where the model was proposed. Click on a model in the Word2Vec section of the page.\n",
    "\n",
    "Note the details section, which describes what the module expects as input, how it preprocesses data, what it does when it encounters a word it hasn't seen before (OOV means \"out of vocabulary\") and in this case, how word embeddings can be composed to form sentence embeddings.\n",
    "\n",
    "Finally, note the URL of the page. This is the URL you can copy to instantiate your module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Task 1: Create an embedding using the NNLM model</h2>\n",
    "\n",
    "To complete this task:\n",
    "<ol>\n",
    "    <li>Find the module URL for the NNLM 50 dimensional English model</li>\n",
    "    <li>Use it to instantiate a module as 'embed'</li>\n",
    "    <li>Print the embedded representation of \"cat\"</li>\n",
    "</ol>\n",
    "\n",
    "NOTE: downloading hub modules requires downloading a lot of data. Instantiating the module will take a few minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.\n",
      "INFO:tensorflow:Downloading TF-Hub Module 'https://tfhub.dev/google/nnlm-en-dim50/1'.\n",
      "INFO:tensorflow:Downloaded TF-Hub Module 'https://tfhub.dev/google/nnlm-en-dim50/1'.\n",
      "INFO:tensorflow:Initialize variable module/embeddings/part_0:0 from checkpoint b'/tmp/tfhub_modules/7f07056e3a4c9f125d5bd920ef3883605d8556a8/variables/variables' with embeddings\n",
      "[[ 0.11233182 -0.3176392  -0.01661182 -0.07224456  0.18654485 -0.13343827\n",
      "   0.14713244 -0.05857142 -0.1187038  -0.09351522  0.3228137  -0.07136346\n",
      "  -0.09905618  0.12076239 -0.13954093 -0.00105632  0.31129223  0.09696656\n",
      "   0.1131188  -0.2877357   0.02663439  0.11532657 -0.1753378  -0.11511131\n",
      "  -0.1570579   0.20969287 -0.15480006 -0.00777875  0.13566507  0.07828814\n",
      "   0.08037776 -0.00566911 -0.17601766  0.04775887  0.0203435  -0.11961495\n",
      "  -0.01772922 -0.02686167 -0.09766243  0.16661587  0.19552311 -0.06655143\n",
      "   0.07954271 -0.14783832 -0.00695672  0.22246888 -0.14037196 -0.12318342\n",
      "  -0.02290071  0.04016034]]\n"
     ]
    }
   ],
   "source": [
    "# Task 1\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "\n",
    "module_url = \"https://tfhub.dev/google/nnlm-en-dim50/1\"\n",
    "embed = hub.Module(module_url)\n",
    "embeddings = embed([\"cat\"])\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    sess.run(tf.tables_initializer())\n",
    "    print(sess.run(embeddings))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When I completed this exercise, I got a vector that looked like `[[ 0.11233182 -0.3176392  -0.0166118...]]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Task 2: Assess the Embeddings Informally</h2>\n",
    "\n",
    "<ol>\n",
    "    <li>Identify some words to test</li>\n",
    "    <li>Retrieve the embeddings for each word</li>\n",
    "    <li>Determine what method to use to compare each pair of embeddings</li>\n",
    "</ol>    \n",
    "\n",
    "So, now we have some vectors but the question is, are they any good? One way of testing whether they are any good is to try them for your task. But, first, let's just take a peak. \n",
    "\n",
    "For our test, we'll need three common words such that two of the words are much closer in meaning than the third."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_1 = \"cat\"\n",
    "word_2 = \"dog\"\n",
    "word_3 = \"potato\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll use the same process of using our Hub module to generate embeddings but instead of printing the embeddings, capture them in a variable called 'my_embeddings'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Message: cat\n",
      "Embedding size: 50\n",
      "Embedding: [0.1123318150639534, -0.31763920187950134, -0.016611823812127113, ...]\n",
      "\n",
      "Message: dog\n",
      "Embedding size: 50\n",
      "Embedding: [0.23500140011310577, -0.24834826588630676, -0.08318911492824554, ...]\n",
      "\n",
      "Message: potato\n",
      "Embedding size: 50\n",
      "Embedding: [0.23423029482364655, -0.34152671694755554, -0.1602797508239746, ...]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Task 2b\n",
    "# Reduce logging output.\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "messages = [word_1, word_2, word_3]\n",
    "def create_embeddings(messages, embed):\n",
    "    my_embeddings = None\n",
    "    with tf.Session() as session:\n",
    "      session.run([tf.global_variables_initializer(), tf.tables_initializer()])\n",
    "      message_embeddings = session.run(embed(messages))\n",
    "      my_embeddings = np.array(message_embeddings)\n",
    "      for i, message_embedding in enumerate(np.array(my_embeddings).tolist()):\n",
    "        print(\"Message: {}\".format(messages[i]))\n",
    "        print(\"Embedding size: {}\".format(len(message_embedding)))\n",
    "        message_embedding_snippet = \", \".join(\n",
    "            (str(x) for x in message_embedding[:3]))\n",
    "        print(\"Embedding: [{}, ...]\\n\".format(message_embedding_snippet))\n",
    "    return my_embeddings\n",
    "my_embeddings = create_embeddings(messages, embed)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we'll use Seaborn's heatmap function to see how the vectors compare to each other. I've written the shell of a function that you'll need to complete that will generate a heatmap. The one piece that's missing is how we'll compare each pair of vectors. Note that because we are computing a score for every pair of vectors, we should have len(my_embeddings)^2 scores. There are many valid ways of comparing vectors. Generality, similarity scores are symmetric. The simplest is to take their dot product. For extra credit, implement a more complicated vector comparison function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py3env/lib/python3.5/site-packages/matplotlib/font_manager.py:1320: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans\n",
      "  (prop.get_family(), self.defaultFamily[fontext]))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAcYAAAF4CAYAAADOolksAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzt3Xt8z/X///H7e2y05RCbOR/KmE8OG3JYIqtRyL5lKZ+QRFPIR5IcsvGhA1+TfEI+JF+HUmpm5SNybMgcJj41cmjIYfRhMcvM9vr98fl593437G17vb297Ha9XF6Xy/Z8P9/P1+O9LY8ez+fz9XrZDMMwBAAAJEleng4AAIBbCYkRAAAHJEYAAByQGAEAcEBiBADAAYkRAAAHJEbcErZu3ar69evr5MmTng7F4+6//37NnTu3SGNkZ2erfv36WrlypenjmBGfJB08eFD169fXnj17ijwWYKaSng4Arrt48aJmzZqlFStW6OTJk7rjjjtUvXp1RUZGqnfv3p4Oz2V/+ctfNGHCBD3xxBP2ttDQUCUlJalixYqFGrN+/frXfb1atWpau3Ztocb+s6lTp+rrr78uctIpiq1bt2rmzJlKTU3V77//Ln9/fzVq1EijRo1SYGCgSpUqpaSkJJUrV65I5zFrnKupXbu2kpKSdNddd0mSDh8+rA4dOmjJkiUKCQkx/XyAq0iMFhIbG6utW7dq9OjRql+/vi5cuKAff/xRx48f93RoRebj46OAgIBCvz8pKcn+9e7du/XSSy/ps88+U5UqVSRJJUqUKHKMt4q9e/fq+eefV48ePTRy5Ej5+vrql19+0bp163ThwgV7v6L8PB2ZNY6jS5cuFfl3DriNActo1qyZsWDBggL7ffnll0bXrl2Nhg0bGu3btzfefPNN48KFC/bXe/bsaYwcOdKIi4szWrVqZTRr1syIi4szcnNzjenTpxutW7c2WrZsacTFxTmNu3z5ciMqKspo2rSp0aJFC6N///7GoUOH7K8fPXrUqFevnvHVV18Z0dHRRuPGjY3w8HAjPj7e3qd9+/ZGvXr1nA7DMIzvvvvOqFevnnHixAl738OHDxuDBw827rvvPqNx48ZGly5djLVr1xb4+bdt22bUq1fPOHr0aL7XsrOzjbi4OOPBBx80GjVqZHTu3NlYunSp/fUtW7YYDRo0MNavX29v27hxo9GgQQNj8+bNxuLFi/PF/8EHHxiGYRhhYWHGnDlznM43bNgwo2/fvvbv169fb/z1r3817rvvPqNZs2ZGr169jH//+99O77naOI4++OAD44EHHrjuz+DixYtGvXr1jH/9619O3y9evNgYNGiQ0aRJE6N9+/bG6tWrjbNnzxovv/yyERISYjz88MPGmjVrrjnO1eL74osvjCeeeMIIDQ01WrZsaURHRxuHDx+2v37gwAH738Vzzz1nNG7c2IiLi7O37969234ex6Njx472Pn/+GV35nTj+vQBmYY3RQgICAvTtt98qIyPjmn2++OILxcbG6rnnntOKFSv0zjvvaPPmzYqJiXHq9/XXX+vy5ctavHixXn/9dc2aNUvR0dHKysrSokWLNGLECM2aNUsbNmywv+fSpUt66aWXFB8fr3nz5snLy0vR0dG6dOmS09hTpkxR165dtXz5cj366KMaNWqU0tLSJElLly5ViRIlNGrUKCUlJTlVeo5Onz6tp59+WufOndOMGTOUmJioIUOGyMuraH+yI0aM0MaNG/Xmm2/qq6++UnR0tCZMmKDly5dLklq1aqUXXnhBr7/+uk6dOqXTp09rxIgR6t+/v1q3bq3HH39czz77rGrVqmWPv2fPni6fPysrS88++6w+/fRTLV68WJUrV1a/fv10/vx5l8cICAjQmTNntHnz5hv+/O+//74iIiK0bNkytWrVSsOHD9ewYcMUHh7u1HYj8eTk5Ojll1/WsmXLNGfOHF2+fFkvvviiLl++7NRv8uTJ6tatm7788ks9+eSTTq+VKlVKS5YskSTNnj1bSUlJWrx4se655x7dd999+vTTT536f/bZZ2rbtq0qV658wz8DoECezsxw3fbt240HH3zQCA4ONrp06WKMGTPGWL16tZGXl2fv0759e2Px4sVO70tOTjbq1atnZGRkGIbx34qxa9euTn06depkdOnSxantscceM95+++1rxnP27FmjXr16xvbt2w3D+KNi/PDDD+19cnJyjJCQEOPjjz+2tzVo0MD4/PPPncb6c8U4depUIywszKnSddW1KsYr1ceRI0ec2qdMmWI8+eST9u8vX75s9OjRw+jdu7fx7LPPGk899ZSRk5Njfz0uLs7o2LFjvvO6UjH+WU5OjtGkSRNj5cqV1x3nz+8ZPny4Ua9ePaNly5ZGv379jH/+85/GyZMn7X2uVTFOnjzZ3uf48eNGvXr1nH7H6enpRr169YxNmzZddRxX4rsyxp49ewzD+OPn/s9//tOpn2PFaBiGkZaWZtSrV89ISUlx6rd8+XKjadOmRlZWlmEYhvGf//zHuPfee50qW8BMrDFaSLNmzbR69Wrt3r1bu3bt0rZt2/Tyyy+rbdu2mjlzps6ePatjx47p7bff1qRJk+zvM/7/feIPHz6sxo0bS5KCg4Odxvb395e/v79TW0BAgP7zn//Yv09NTdU//vEPpaam6uzZs/b248ePq1mzZvbvHccuWbKkKlasqF9//fWGPusPP/yg0NBQ+fr63tD7rufK7seuXbs6tV++fNnpPCVKlNCUKVPUqVMneXl5KTExUSVLmvOfSlpamqZPn67vv/9eZ86ckWEYunjx4g2tE5csWVKTJk3SsGHDtHXrVu3Zs0eLFy/WjBkzNGfOHDVt2vSa73X83Vz5fTtuXLqy5uf4ey/Iv//9b73//vvat29fvr+Lhg0b2r+/8rd3ozp27KiJEydqxYoV6tatm+Lj41WhQgW1a9euUOMBBSExWkzJkiXVtGlTNW3aVH379lVCQoJee+01bdu2TXfffbckafTo0WrZsmW+9zpOO/35H3qbzSZvb+9878nLy5Mk/f777+rbt6+aNWumN9980/4PaOfOnZWTk+P0nj+PY7PZ7Mn5Rthstht+z/UYhiGbzaalS5fm+/x/nqL94YcflJ2dLZvNpvT0dFWtWtWleP/8Of88ndi/f39VrVpV48aNU2BgoLy9vRUVFZXvZ+iKwMBAde3aVV27dtWrr76qxx57zJ4cr8Xxc1/5+Tr+vq60Xfm9F+T8+fPq27evwsLC9Pbbb6tixYrKyclRZGRkvs90xx13uPzZHPn4+Ojxxx/XZ599pm7dumnp0qXq1q3bbbWhCrcWEqPF3XPPPZL++3/4LVq0UJUqVfTzzz+re/fupp7n4MGDOnPmjIYOHWo/586dOwuV8Ly9vZWbm3vdPvfee68+++wzZWVlmVY1NmzYUIZhKD09XWFhYdfsd/LkSY0ePVqDBw/WqVOnNGzYMC1btkxly5a1x3+1xFGxYkWdOnXK/r1hGEpNTVX16tUlSenp6Tpy5Ij+/ve/q1WrVpKko0eP6ty5c0X+bKVKlVK1atV05syZIo91I3766Sf99ttvGjZsmGrUqCFJ2rJlS6HGupKgr/a38dRTT2nevHn6+OOPlZaWpqioqMIHDRSAzTcW0rNnT3388cfas2ePjh07pi1btmjcuHEqW7asvUL829/+pgULFmjGjBn66aefdOjQIX3zzTcaO3Zskc5dtWpV+fj4aMGCBTpy5Ii2bNmiiRMnFqqqq169urZu3ar09PRr/kP+17/+VXl5eXrppZe0Y8cOHT16VOvWrXPaDHSjgoKC1KVLF73++utKTEzUkSNHlJqaqs8++0wffvihpP/+o/zqq6+qfv36io6O1uuvvy4/Pz+NGTPGKf4TJ05oz549OnPmjC5evChJCgsL0/Lly7VlyxYdPHhQ48ePd5pCrlChgsqWLaslS5YoLS1NO3bs0PDhw1WqVKkb+hwLFixQbGyskpKSdOTIER08eFAzZszQd999p4cffrjQP5/CqF69ury9vbVgwQIdPXpUSUlJmjx5cqHG8vf3t183+euvvzr9D0Pt2rXVsmVLTZw4Uffff7+qVatm1kcA8iExWkjbtm2VmJioF154QY888ohGjhypWrVq6eOPP1aFChUkSf/zP/+jd999Vxs2bNCTTz6pqKgoTZ8+XZUqVSrSuStUqKDJkydr8+bN6ty5s9555x2NGDGiULtER4wYoR9++EEPPfSQWrdufdU+lSpV0uLFi+Xn56cXXnhBXbp00dSpU4v0GSTpnXfeUY8ePfSPf/xDnTp10nPPPafly5erZs2akqSZM2fqwIEDmjx5sry8vFSqVClNnTpVGzZssO+afPTRRxUeHq7nn39erVu31v/93/9Jkl588UWFhYVp8ODB6t27typVqqT27dvbz+3t7a1p06Zp//79euyxx/TGG2+of//+Kl++/A19hiZNmujChQuKjY1Vly5d1KNHD61Zs0Zjx47Viy++WOSf0Y0IDAzU22+/rbVr16pTp06Ki4vTyJEjCzWWj4+P3njjDSUkJKht27b5Zj2eeuop5eTk6KmnnjIjdOCabEZh5sIA4CabN2+e5s6dq/Xr15u2GQq4Gv66ANzSMjMzlZaWpo8++ki9e/cmKcLtmEoFcEt744039PTTT+vee+9Vnz59PB0ObgELFy7UE088oYYNG+r111+/bt+PPvpI999/v5o1a6aRI0fmuyHJ1TCVCgCwlFWrVsnLy0vffvutsrOz9fbbb1+137fffqsRI0Zo/vz5qlSpkgYNGqQmTZro1Vdfve74VIwAAEvp0KGDHn744QI3ri1btkxRUVEKCgpSuXLl7Le0LAiJEQBwW9q/f7/T3Z7q16+vX3/91ekOTVdzU1exx9mu/8w8WFffKlmeDgFuUn16I0+HADexdVvhtrGL8u99jLHPlBiysrJ055132r8vU6aMJOnChQv254BeDdu7AACmuxWmI319fZWZmWn//srXfn5+133frRA7AACmCwoK0r59f1Sfe/fulb+//3WrRYnECABwA68iHAW5fPmysrOzlZeXp9zcXGVnZ+e7Yb8kRUZGaunSpTpw4IB+++03zZw5U48//rhLsQMAYCp3JsaZM2eqcePGmj17tpYvX67GjRtr5syZOn78uEJDQ+2PcWvbtq369eun3r17q3379qpWrZpefvnlAse/qdcxsvnm9sXmm9sXm29uX+7cfPNOEf69H2HS5pvCYvMNAMB0Vp6OJDECAExn7mPGby4SIwDAdFauGK0cOwAApqNiBACYzspVF4kRAGA6EiMAAA5IjAAAOCAxAgDggMQIAIADKydGK8cOAIDpqBgBAKazctVFYgQAmI7ECACAAxIjAAAOSIwAADjg6RoAADiwcsVo5dgBADAdFSMAwHRWrrpIjAAA05EYAQBwQGIEAMABiREAAAckRgAAHFg5MVo5dgAATEfFCAAwnZWrLhIjAMB0JEYAAByQGAEAcMBNxAEAcEDFCACAAysnRivHDgCA6agYAQCms3LVRWIEAJjOZuHdNyRGAIDpvGyGp0MoNBIjAMB0VIwAADiwcF4kMQIAzGez8FSqlTcOAQBgOipGAIDpWGMEAMCBlROjy1Opc+fOvWr7vHnzTAsGAHB78LIZhT48zeXE+P7771+1febMmaYFAwC4PdiKcHhagVOpW7ZskSTl5eXpu+++k2H8kc1/+eUX+fn5uS86AIAlWXkqtcDEOHr0aElSdna2Ro0aZW+32WwKCAjQmDFj3BcdAMCSbuvEuHbtWknSa6+9pkmTJrk9IAAAPMnlXakkRQCAq6x8gb/LiTEzM1PTp0/Xtm3bdPbsWae1xvXr17sjNgCARXlZeCrV5V2psbGx+vHHH/XSSy8pIyNDY8aMUZUqVdSnTx83hgcAsCKbrfCHp7lcMW7atEkrVqzQXXfdpRIlSujhhx9Wo0aNNGDAAJIjAMCJTdadSnW5YszLy1OZMmUkSb6+vjp37pwCAgJ0+PBhtwUHALAmd1aMGRkZGjhwoEJCQtS+fXslJiZetd+lS5c0duxYhYWFqUWLFhowYIDS09MLHN/lxBgcHKxt27ZJkpo3b65x48YpNjZWtWvXdnUIAEAx4c7EOH78eHl7e2vTpk2aPHmyYmNjtX///nz95s+fr127dmn58uX69ttvVaZMGf39738vcHyXE+OECRNUvXp1SdKYMWNUunRpnT9/XpMnT3Z1CAAAiiQrK0urVq3SkCFD5Ofnp+bNmys8PFwJCQn5+v7yyy9q06aN/P39VapUKXXu3PmqCfTPXE6M8+fP1+nTpyVJFSpU0MSJE9WrVy998sknN/CRAADFgbvulZqWliYvLy/VqVPH3hYcHKwDBw7k6xsVFaWdO3cqPT1dv//+uxITE9W2bduCY3f1Q3755Zdq2LChU1vDhg315ZdfujoEAKCYcNdUalZWln2/yxVlypTRhQsX8vWtU6eOqlatqrZt26pZs2Y6ePCgBg4cWGDsLidGm83mdO2iJOXm5iovL8/VIQAAxYS7biLu6+urzMxMp7bMzMyr3rc7JiZG2dnZ2rp1q3bt2qWIiAj179+/wNhdTozNmzfXu+++a0+EeXl5mj59upo3b+7qEACAYsJmMwp9XE/t2rWVm5urtLQ0e9vevXtVt27dfH337dunxx9/XOXLl5ePj4969eql3bt368yZM9c9h8uJcfTo0dq8ebPatGmjqKgoPfDAA9q8ebPeeOMNV4cAABQT7ppK9fX1VUREhN577z1lZWVpx44dWrNmjSIjI/P1bdSokRISEnT+/Hnl5ORo8eLFqlSpkipUqHDdc7h8gX/lypUVHx+v3bt368SJE6pSpYoaN24sLy+XcysAoJhw5y3hYmJiNGrUKIWFhal8+fKKjY1VUFCQtm/frv79+yslJUXSfx9+MWHCBHXo0EE5OTkKCgq65rOFHdmMPy8cutE4W/2bdSrcZH2rZHk6BLhJ9emNPB0C3MTWbYXbxv7Jv3ah31vv1zTT4igMlytGAABcVSyergEAgKtugXuBFxqJEQBgulvhKRmFRWIEAJiOqVQAABxY+UHFJEYAgOmsPJXKRYgAADigYgQAmM7KFSOJEQBgOpvYfAMAgB0VIwAADmwW3pZKYgQAmM5m4a2dJEYAgOmsPJVq4ZwOAID5qBgBAOZjjREAgD+wxggAgAObhRcZSYwAANNRMQIA4IiKEQCAP1i5YrRw6AAAmI+KEQBgOm4JBwCAAwsvMZIYAQDms/IaI4kRAGA+plIBAPgDU6kAADiw8uYbC88CAwBgPipGAIDp2HwDAIADbiIOAIAjKkYAAP5g4YKRxAgAMJ+Vd6WSGAEApmPzjYv6Vsm6mafDTfThCV9PhwA3GXuHj6dDAG4qKkYAgPksvMhIYgQAmI6pVAAAHLD5BgAABxaeSSUxAgDMR8UIAIAj6+ZFK9+0BwAA81ExAgBMx65UAAAcsMYIAIADdqUCAOCAihEAAEesMQIA4MDCFaOFczoAAOajYgQAmM/CZZeFQwcA3LK8bIU/CpCRkaGBAwcqJCRE7du3V2Ji4jX7/vDDD3rmmWcUGhqqsLAwzZ8/v8DxqRgBAOZzY9k1fvx4eXt7a9OmTUpNTVV0dLSCg4MVFBTk1O/MmTPq16+fRo4cqUceeUSXLl1Senp6geNTMQIAzOemijErK0urVq3SkCFD5Ofnp+bNmys8PFwJCQn5+n700Udq06aNunbtKh8fH91555265557Cg690B8aAIBrcVNiTEtLk5eXl+rUqWNvCw4O1oEDB/L13bVrl8qVK6enn35arVu31oABA3T8+PGCQ7/xTwsAQAG8inBcR1ZWlsqUKePUVqZMGV24cCFf3/T0dC1btkyjRo3S+vXrVb16db3yyisuhQ4AgCX4+voqMzPTqS0zM1N+fn75+pYqVUoRERFq3LixSpUqpYEDByolJUXnz5+/7jlIjAAA87lpKrV27drKzc1VWlqavW3v3r2qW7duvr7169d3+t72/2/gahjG9UN38SMCAOA6N02l+vr6KiIiQu+9956ysrK0Y8cOrVmzRpGRkfn6PvHEE/rmm2+UmpqqnJwczZgxQ82aNVPZsmULDB0AAHO58TrGmJgYXbx4UWFhYRo2bJhiY2MVFBSk7du3KzQ01N6vdevWGjp0qF544QWFhYXpyJEjmjJlSoHj24yCakoTHa1a42adCjfZhyd8PR0C3GTsVw08HQLcxNZpmdvGzu0fWnCnayjxzxQTI7lxXOAPADCfhW8iTmIEAJjPwomRNUYAABxQMQIAzGfhsovECAAwn4WnUkmMAADT2agYAQBwQMUIAIADC1eMFg4dAADzUTECAMzHVCoAAA5IjAAAOLDwQh2JEQBgPipGAAAcUDECAODAwhWjhXM6AADmo2IEAJjPwmUXiREAYD4LT6WSGAEA5qNiBADAARUjAAAOikNiXLp06VXbfXx8VLlyZYWEhMjHx8e0wAAAFlYcplITEhKUkpIif39/Va5cWSdPntSvv/6qhg0b6tixY5KkGTNmqFGjRm4LFgAAd3M5MdatW1cRERHq3bu3vW3hwoU6dOiQPv74Y82cOVMTJkzQkiVL3BIoAMBCLDyV6nKx++WXX6pnz55ObT169FBiYqJsNpv69eunAwcOmB4gAMCCvIpweJjLIVSsWFFr1651alu/fr0qVKggScrOzlbJkuzlAQBIstkKf3iYy5lszJgxGjJkiIKCglSlShWdOHFC+/fv17Rp0yRJ33//vXr16uW2QAEAFuL5/FZoLifGNm3aaPXq1dq4caNOnTqldu3aqV27drrrrrvsr7dp08ZtgQIALOQWqPwK64bmPitUqKAWLVooPT1dgYGB9qQIAIAT6+ZF1xPjqVOn9Morr2jXrl0qX768MjIy1KRJE8XFxSkwMNCdMQIAcNO4vPkmNjZWwcHBSk5OVlJSkpKTk9WgQQPFxMS4Mz4AgBUVh803O3bs0LRp0+Tt7S1J8vX11WuvvaYHHnjAbcEBACzqFrjsorBcDr1cuXI6ePCgU9uhQ4dUtmxZ04MCAFhccagY+/Xrpz59+igqKkpVq1bVsWPHFB8fryFDhrgzPgCAFXk+vxWay4mxe/fuqlmzphITE7Vv3z4FBgZqypQpat26tTvjAwBY0S1Q+RXWdRPjlYv3HVWqVEkBAQGy2WxKTk5WcnIyVSMAwJl18+L1E+PJkyftX2dnZ2vVqlVq2LChqlWrpuPHj2vPnj3q0KGD24MEAOBmuW5ifOutt+xfDx06VFOmTFHHjh3tbatWrdLKlSvdFx0AwJosPJXq8q7UjRs36uGHH3Zqe+ihh7RhwwbTgwIAWFxxeLpGrVq1tGjRIqe2xYsXq2bNmqYHBQCwuOJwucaECRM0aNAgzZkzR4GBgUpPT1fJkiU1ffp0d8YHALAiz+e3QnM5Mf7lL3/R119/re+//16nTp1SQECAQkJC7HfCAQDA7hao/Arrhp6u4e3trebNm7srFgDAbcLCefFWWOYEAODWcUMVIwAALrFwyUhiBACYz7p5kcQIAHADL+tmRhIjAMB81s2LJEYAgBuwxggAgAPr5kUu1wAAWEtGRoYGDhyokJAQtW/fXomJidftf+nSJT3yyCNq27atS+NTMQIAzOfGqdTx48fL29tbmzZtUmpqqqKjoxUcHKygoKCr9p87d64qVqyorKwsl8anYgQAmM9WhOM6srKytGrVKg0ZMkR+fn5q3ry5wsPDlZCQcNX+R48e1fLly/XCCy+4HDqJEQBgPi9b4Y/rSEtLk5eXl+rUqWNvCw4O1oEDB67af8KECXrllVdUunRp10N3uScAAK5yY8VYpkwZp7YyZcrowoUL+fquXr1aly9fVkRExA2FzhojAMB8blpj9PX1VWZmplNbZmam/Pz8nNqysrI0efJkzZ49+4bPQWIEAJjPTXtvateurdzcXKWlpal27dqSpL1796pu3bpO/Q4fPqxjx47pmWeekSTl5OTo/Pnzuv/++7VkyRJVr179mucgMQIALMPX11cRERF67733NGHCBKWmpmrNmjX65JNPnPoFBQVp/fr19u9TUlI0fvx4xcfHq0KFCtc9B2uMAADz2WyFPwoQExOjixcvKiwsTMOGDVNsbKyCgoK0fft2hYaGSpJKliypgIAA+1GuXDl5eXkpICBAJUqUuH7ohmEYpvwQXHC0ao2bdSrcZB+e8PV0CHCTsV818HQIcBNbp2VuG9tI6FLo99oivzQxkhvHVCoAwHw8XQMAAAfcRBwAAAckRgAAHFg4MbIrFQAAB1SMAADz2axbd5EYAQDmY1cqAAAOLLzGSGIEAJiPqVQAABxQMQIA4MDCa4zWrXUBAHADKkYAgPlYYwQAwAFrjK6pPr3RzTwdbqKxd/h4OgS4yfjOqZ4OAW4S486HDpIYAQBwwFQqAAAOLLwrlcQIADCfhadSrVvrAgDgBlSMAADzscYIAIADC0+lkhgBAOZj8w0AAA6YSgUAwAFTqQAAOLBwYrRurQsAgBtQMQIAzGfhipHECAAwn5d1JyRJjAAA81ExAgDggMQIAIADrmMEAMCBhe98Y92UDgCAG1AxAgDMxxojAAAOWGMEAMABFSMAAA5IjAAAOODONwAAOLJuxWjdlA4AgBtQMQIAzMcaIwAADrhcAwAAR1SMAAD8galUAAAcMJUKAIAj61aM1k3pAAC4ARUjAMB8rDECAODIuhOS1o0cAHDrstkKfxQgIyNDAwcOVEhIiNq3b6/ExMSr9pszZ466dOmi0NBQhYeHa86cOS6FTsUIADCfG6dSx48fL29vb23atEmpqamKjo5WcHCwgoKCnPoZhqF33nlH9evX15EjR/T888+rSpUq6ty583XHp2IEALiBrQjHtWVlZWnVqlUaMmSI/Pz81Lx5c4WHhyshISFf3/79++vee+9VyZIldffdd+uhhx7Szp07C4ycxAgAMJ/Nq/DHdaSlpcnLy0t16tSxtwUHB+vAgQPXfZ9hGNq+fbvq1q1bYOgkRgCAZWRlZalMmTJObWXKlNGFCxeu+77p06crLy9P3bp1K/AcrDECAMznpjVGX19fZWZmOrVlZmZaRpF/AAAQHElEQVTKz8/vmu9ZuHChli1bpsWLF8vHx6fAc1AxAgDcwD1rjLVr11Zubq7S0tLsbXv37r3mFOnSpUs1e/ZszZ8/X5UrV3YpchIjAMB8blpj9PX1VUREhN577z1lZWVpx44dWrNmjSIjI/P1Xb58uaZOnap58+apRo0aLodOYgQAmM5msxX6KEhMTIwuXryosLAwDRs2TLGxsQoKCtL27dsVGhpq7/fuu+8qIyNDUVFRCg0NVWhoqMaOHVtw7IZhGEX69DfA+LzTzToVbrY7Cp63hzWN75zq6RDgJjHGPvcNfiyu8O+t9op5cRQCm28AAOaz8GOnrBs5AABucEMV42+//aZ169YpPT1dgYGBevDBB1W+fHl3xQYAsCzrPl3D5YoxJSVFERER+uSTT7Rv3z598skn6tChg1JSUtwZHwDAitx4E3F3c7lifPPNNxUTE+N089UVK1ZowoQJ+vzzz90SHADAoorDGmNaWpoeffRRp7aOHTvqyJEjpgcFALA691zgfzO4nBhr1aqlr776yqlt5cqVN3TRJACgmCgOU6mjRo3SgAEDtGDBAlWtWlXHjh3T4cOHNWvWLHfGBwCwIgtPpbqcGJs2barVq1dr/fr1OnXqlNq3b6927dqxKxUAcFtxOaVPmDBB5cqVU2RkpPr376/IyEiVL19eEydOdGd8AABLKgZrjF988cVV25cvX25aMACA28TtvMa4dOlSSVJubq796yuOHj3KVCoA4Cpu4zXGhIQESVJOTo79a+m/d0739/fXO++8477oAADWdAtUfoVVYGJcsGCBJGnq1KkaOnSo2wMCANwGbufEeIVjUjQMQ45Pq/Lysm7JDABwB+vmBZcTY3p6usaPH6/t27fr3LlzTq+lpvK8NgDA7cHllB4TEyNvb2999NFH8vX1VXx8vMLDwzVu3Dh3xgcAsKLbeVfqFSkpKVq3bp18fX1ls9kUHBysiRMn6umnn1b37t3dGSMAwHI8n+AKy+XE6OXlpZIl/9u9bNmyOnPmjO68806lp6e7LTgAgEUVh1vCNWnSRBs2bFBERITatGmjv/3tbypdurQaNmzozvgAAFZ0C0yJFpbLiXHSpEnKy8uT9N8bin/44Ye6cOGC+vTp467YAACWZd3E6HKtu2nTJvtdbkqXLq2XXnpJw4cP144dO9wWHADAomxehT88zOUIRo8efdX2sWPHmhYMAACeVuBU6tGjRyX996L+K187vubj4+OeyAAAFmbdqdQCE2NERIRsNpsMw1BERITTa/7+/ho8eLDbggMAWNTtvPlm7969kqSePXtq4cKFbg8IAHA78PxaYWG5vCv1SlI8fvy40tPTVblyZVWpUsVtgQEALOx2rhivOH36tIYOHapdu3apfPnyysjIUJMmTRQXF6fAwEB3xggAsJpbYHdpYd3QvVKDg4OVnJyspKQkJScnq0GDBoqJiXFnfAAAS7IV4fAslyvGHTt2aNq0afL29pYk+fr66rXXXtMDDzzgtuAAALjZXK4Yy5Urp4MHDzq1HTp0SGXLljU9KACAxRWHp2v069dPffr0UVRUlKpWrapjx44pPj5eQ4YMcWd8AABLsu4ao8uJsXv37qpZs6YSExP1008/qVKlSpoyZYpat27tzvgAAFZ0C1R+heVyYrx06ZK2bt2q5ORknTp1SoGBgfL391fTpk1VqlQpd8YIALCcYlAxxsbG6ueff9aYMWNUrVo1HT9+XB988IHS09P11ltvuTNGAIDVFIeKcc2aNVq9erV9s03dunXVuHFjdejQwW3BAQAsysKJ0eVa19/fX7///rtTW3Z2tgICAkwPCgAAT3G5YoyMjFS/fv3Uq1cvBQYG6uTJk1q0aJEiIyO1ZcsWez824wAArLzGaDMMw3ClY3h4eMGD2Wxas2bNNV83Pu/kemSwljt4/NjtanznVE+HADeJMfa5b/Ds1YV/b6mIgvu4kcsV49q1a90ZBwDgtmLdNUaXEyMAAC6z8E3ESYwAADegYgQA4A8WrhitGzkAAG5AxQgAcAOmUgEA+IOF73xDYgQAmM/Ca4wkRgCAG1AxAgDwBwtPpVq31gUAwA1IjAAAN/AqwnF9GRkZGjhwoEJCQtS+fXslJiZetZ9hGJo8ebJatmypli1batKkSXLl9uBMpQIAzOfGqdTx48fL29tbmzZtUmpqqqKjoxUcHKygoCCnfkuWLNE333yjhIQE2Ww2Pffcc6pRo4Z69Ohx3fGpGAEAbuCeijErK0urVq3SkCFD5Ofnp+bNmys8PFwJCQn5+i5btkx9+/ZV5cqVFRgYqOeee07x8fEuRQ4AgLlstsIf15GWliYvLy/VqVPH3hYcHKwDBw7k67t//34FBwc79du/f3+Bod/UqVRbtxU383QATBDj0hNbgT9r5pZRs7KyVKZMGae2MmXK6MKFC1fte+eddzr1y8rKkmEYsl0nAVMxAgAsw9fXV5mZmU5tmZmZ8vPzu2pfx4SZmZkpX1/f6yZFicQIALCQ2rVrKzc3V2lpafa2vXv3qm7duvn6BgUFae/evU79/rxB52pIjAAAy/D19VVERITee+89ZWVlaceOHVqzZo0iIyPz9Y2MjNS8efOUnp6u9PR0zZs3T48//niB57AZrlzUAQDALSIjI0OjRo3S5s2bVb58eQ0bNkyPPfaYtm/frv79+yslJUXSH9cxLl26VJIUFRWl4cOHFziVSmIEAMABU6kAADggMQIA4IDECACAAxKjiebOnXvV9nnz5t3kSAAAhcXmGxM1bdpUO3fuzNfeokULJScneyAimOXKrrY/8/HxUeXKlRUSEiIfH5+bHBXM8ttvv2ndunVKT09XYGCgHnzwQZUvX97TYcFDeLqGCbZs2SJJysvL03fffef0WJNffvnlqndkgLUkJCQoJSVF/v7+qly5sk6ePKlff/1VDRs21LFjxyRJM2bMUKNGjTwcKW5USkqKoqOjdffdd6tq1apat26d3nzzTX3wwQcKDQ31dHjwACpGE4SHh0uSTpw4oSpVqtjbbTabAgIC1L9/fz300EOeCg8mGDdunOrUqaPevXvb2xYuXKhDhw7pjTfe0MyZM7VhwwYtWbLEg1GiMJ588kn16dNHnTt3tretWLFCc+fO1eeff+7ByOApJEYTvfbaa5o0aZKnw4Ab3Hfffdq6dau8vP5Yls/NzVWrVq20bds2Xbp0Sa1bt9aOHTs8GCUKo6DfLYofNt+YiKR4+6pYsaLWrl3r1LZ+/XpVqFBBkpSdna2SJVmZsKJatWrpq6++cmpbuXKlatSo4aGI4GlUjCbKzMzU9OnTtW3bNp09e9ZprXH9+vWeCwxFlpSUpCFDhigoKEhVqlTRiRMntH//fk2bNk1t2rRRUlKSdu3apUGDBnk6VNygnTt3asCAAapdu7aqVq2qY8eO6fDhw5o1a5aaNm3q6fDgASRGE7366qtKT0/Xs88+q+HDh2vy5MmaO3euOnbsqD59+ng6PBTRmTNntHHjRp06dUqVKlVSu3btdNddd3k6LJjgt99+0/r1651+t+xKLb5IjCZq3bq1VqxYobvuukvNmzfX9u3blZ6ergEDBig+Pt7T4cEEx48ft2/pr1q1qqfDgQkmTJigMWPG5GufOHGiRo8e7YGI4GmsMZooLy/P/mRpX19fnTt3TgEBATp8+LCHI0NRnTp1Sj179lSHDh00ePBgdejQQc8884zS09M9HRqK6Isvvrhq+/Lly29yJLhVsFvARMHBwdq2bZtat26t5s2ba9y4cfLz81Pt2rU9HRqKKDY2VsHBwZo9e7Z8fX2VlZWluLg4xcTEaNasWZ4OD4Vw5aYNubm5+W7gcPToUaZSizGmUk109OhRSVKNGjV05swZTZkyRVlZWRo0aJDuueceD0eHomjZsqWSkpLk7e1tb7t06ZIeeOABbd261YORobB69eolSdqxY4eaNWtmb7fZbPL391fv3r0VEhLiqfDgQVSMJpo/f746deqkGjVqqEKFCpo4caJ27typTz75hLUKiytXrpwOHjyo4OBge9uhQ4dUtmxZD0aFoliwYIEkaerUqRo6dKiHo8GthIrRRK1atdLGjRud7pl56dIltWvXzn7bOFjTp59+qri4OEVFRdm39MfHx2vIkCF66qmnPB0eTGIYhtNlVo4X/aP4oGI0kc1m05//PyM3N1d5eXkeighm6d69u2rWrKnExETt27dPgYGBmjJlilq3bu3p0FBE6enpGj9+vLZv365z5845vZaamuqhqOBJVIwmGjx4sKpXr67hw4fLy8tLeXl5+t///V8dPnxY77//vqfDww2aNm3aVdsNw5DNZrN/P2TIkJsVEtxgwIABKl26tKKjo9WzZ08tWrRI06dPV7t27dS9e3dPhwcPoGI00ejRoxUdHa02bdqoatWqOnHihAICAti1aFEnT560f52dna1Vq1apYcOGqlatmo4fP649e/aoQ4cOHowQZkhJSdG6devk6+srm82m4OBgTZw4UU8//TSJsZgiMZqocuXKio+P1+7du+1P2mjcuDHrFBb11ltv2b8eOnSopkyZoo4dO9rbVq1apZUrV3oiNJjIy8vLfp/bsmXL6syZM7rzzju5RrUYYyoVcEGzZs2UnJysEiVK2Ntyc3PVokULnqhhcQMGDFC3bt0UERGhsWPHKi0tTaVLl9bvv/9u37mK4oVSBnBBrVq1tGjRIqe2xYsXq2bNmh6KCGaZNGmS7rvvPknSqFGj1KpVKwUFBSkuLs7DkcFTqBgBF/z4448aNGiQLl++rMDAQKWnp6tkyZKaPn267r33Xk+HhyL417/+pUcffTRf+8qVK/XII494ICJ4GokRcFFOTo6+//57nTp1SgEBAQoJCXG6Ew6sqWnTptq5c2e+9hYtWig5OdkDEcHT2HwDuMjb21vNmzf3dBgwyZVbOBqGYf/a8TXHG3WgeKFiBFAsBQcHX/WmHJLk7++vwYMHc1ejYorECKBY69mzpxYuXOjpMHALITECgP54CHXlypVVpUoVT4cDD2KNEUCxdvr0aQ0dOlS7du1S+fLllZGRoSZNmiguLk6BgYGeDg8ewHWMAIq1mJgYBQcHKzk5WUlJSUpOTlaDBg0UExPj6dDgIUylAijWeAg1/oyKEUCxduUh1I54CHXxxhojgGKtX79+6tOnz1UfQo3iialUAMXed999p8TERJ0+fVqVKlVS586deQh1MUbFCKBYu3TpkrZu3ark5GSdOnVKgYGB8vf3V9OmTVWqVClPhwcPoGIEUKyNGjVKP//8swYMGGB/CPUHH3ygmjVrOj2TE8UHiRFAsdayZUutXr3aabNNRkaGOnTowE3Eiyl2pQIo1vz9/fX77787tWVnZysgIMBDEcHTqBgBFGuzZ89WYmKievXqpcDAQJ08eVKLFi1Sly5d1KhRI3s/NuMUHyRGAMVaeHh4gX1sNpvWrFlzE6LBrYDECACAA9YYAQBwQGIEAMABiREAAAckRgAAHJAYAQBw8P8A3q+3o01MIHcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7ce69afe10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_similarity(labels, embeddings):\n",
    "  corr = np.inner(embeddings, embeddings)\n",
    "  sns.set(font_scale=1.2)\n",
    "  g = sns.heatmap(\n",
    "      corr,\n",
    "      xticklabels=labels,\n",
    "      yticklabels=labels,\n",
    "      vmin=0,\n",
    "      vmax=1,\n",
    "      cmap=\"YlOrRd\")\n",
    "  g.set_xticklabels(labels, rotation=90)\n",
    "  g.set_title(\"Semantic Textual Similarity\")\n",
    "\n",
    "plot_similarity([word_1, word_2, word_3], my_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What you should observe is that, trivially, all words are identical to themselves, and, more interestingly, that the two more similar words have more similar embeddings than the third word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Task 3: From Words to Sentences</h2>\n",
    "\n",
    "Up until now, we've used our module to produce representations of words. But, in fact, if we want to, we can also use it to construct representations of sentences. The methods used by the module to compose a representation of a sentence won't be as nuanced as what an RNN might do, but they are still worth examining because they are so convenient.\n",
    "\n",
    "<ol>\n",
    "    <li> Examine the documentation for our hub module and determine how to ask it to construct a representation of a sentence</li>\n",
    "    <li> Figure out how the module takes word embeddings and uses them to construct sentence embeddings </li>\n",
    "    <li> Construct a embeddings of a \"cat\", \"The cat sat on the mat\", \"dog\" and \"The cat sat on the dog\"  and plot their similarity\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Message: cat\n",
      "Embedding size: 50\n",
      "Embedding: [0.1123318150639534, -0.31763920187950134, -0.016611823812127113, ...]\n",
      "\n",
      "Message: The cat sat on the mat\n",
      "Embedding size: 50\n",
      "Embedding: [0.25838735699653625, 0.03262478858232498, 0.19812637567520142, ...]\n",
      "\n",
      "Message: dog\n",
      "Embedding size: 50\n",
      "Embedding: [0.23500140011310577, -0.24834826588630676, -0.08318911492824554, ...]\n",
      "\n",
      "Message: The cat sat on the dog\n",
      "Embedding size: 50\n",
      "Embedding: [0.2746334373950958, -0.024848347529768944, 0.17018628120422363, ...]\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/envs/py3env/lib/python3.5/site-packages/matplotlib/font_manager.py:1320: UserWarning: findfont: Font family ['sans-serif'] not found. Falling back to DejaVu Sans\n",
      "  (prop.get_family(), self.defaultFamily[fontext]))\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHbCAYAAADI5J4oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3XtYVNX+P/D3DDcBUeQqqCc9yuXkjcsgggYHFNMQqaTMo9JRA0lUUjSveSEwjZSOJpCXzOp4SU2BjrfUDFFTURS+CiUWioJwDFFg5L5/f/RzTiMoyGYahnm/nmeeh1l7zdqfPaDzmc9ae2+JIAgCiIiIiLSUVN0BEBEREakTkyEiIiLSakyGiIiISKsxGSIiIiKtxmSIiIiItBqTISIiItJqTIaISC3Onj0LBwcH3LlzR92hqN2QIUOwZcsWUWNUVVXBwcEBhw4davVxWiM+ALh+/TocHByQlZUleiyi1qSr7gCI6MkqKyuRmJiIAwcO4M6dOzA0NET37t0RGBiI4OBgdYfXbM8//zyio6Px6quvKtqcnZ2RlpYGc3PzFo3p4ODw1O3dunXD8ePHWzT24+Li4nD48GHRiYYYZ8+eRUJCArKzs/Hw4UNYWFigf//+WLRoEaytrWFgYIC0tDR07txZ1H5aa5zG9OzZE2lpaejSpQsA4MaNGxgxYgR27doFJyenVt8fUXMxGSJqw5YvX46zZ89i8eLFcHBwQEVFBa5evYqCggJ1hyaavr4+LC0tW/z6tLQ0xc+ZmZmYPn06du/eDRsbGwCAjo6O6BjbipycHEydOhXjx4/HwoULYWRkhFu3buH7779HRUWFop+Y9/OPWmucP6qurhb9OydSFU6TEbVhR48exdSpUzF8+HD06NEDjo6OePXVVzFjxgylfv/5z38QGBiI/v37w9fXFx988AHkcrli+6RJk7Bo0SLExcXBw8MDMpkMcXFxqK+vxyeffAJPT08MHjwYcXFxSuOmpKTgtddeg6urK9zd3REaGopff/1Vsf3WrVtwcHDAgQMHEBYWhoEDB2LYsGHYv3+/oo+vry/q6uqwcOFCODg4KCo6jU2T3bx5E7NmzcKgQYMwcOBABAQE4Pvvv2/0vbG0tFQ8HlUxzMzMFG1mZmYAfv8QjouLg4+PDwYMGIDRo0dj7969inF+/PFHPP/88/jhhx8UbSdPnsTzzz+PM2fOYMeOHUhMTMSvv/6qiH/jxo0AGp8+mjt3LqZOnap4/sMPP2DChAkYNGgQZDIZgoODceXKlUaP6UlSU1NhZmamSIp79OgBDw8PLFq0CH/9618BNJzeevR8x44dmDlzJpycnODr64ujR4+itLQUERERcHZ2hp+fn1IFrTnTbfv27cPYsWPh4uKCwYMHIywsDDdv3lRsfzQdduDAAUyZMgUDBw7Ehg0blKbJqqqqMGLECADAuHHj4ODggJEjRyr6PP4ePfqdcFqVVIHJEFEbZmlpiZMnT6K0tPSJfb755hssX74ckydPxoEDB7B69WqcPn0ay5YtU+p3+PBh1NbWYvv27ViwYAESExMxbdo0yOVy/Pvf/8b8+fORmJiolBRUV1dj+vTp2LdvH7Zu3QqpVIpp06ahurpaaew1a9ZgzJgxSE5OxqhRo7Bo0SLk5eUBAPbs2QMdHR0sWrQIaWlpShWdP/rvf/+LN954Aw8ePEB8fDxSUlIQEREBqVTcf1Pz589HamoqVq5cif/85z+YNm0aoqOjkZycDAAYPHgwQkNDsWDBAhQXF+O///0v5s+fj5CQEHh4eOCVV17Bm2++ieeee04R/8SJE5u9f7lcjjfffBNff/01tm/fjq5du+Ktt95CWVlZs8ewtLRESUkJTp8+/czHv2HDBvj5+WH//v0YPHgw5s2bh8jISPj6+iq1PUs8NTU1mDVrFvbv34/NmzejtrYWb7/9Nmpra5X6xcbGYuzYsfj222/x2muvKW0zMDDArl27AAAbN25EWloatm/fjt69e8PNzQ1ff/21Uv/du3fDy8sLXbt2feb3gKhJAhG1Wenp6cLf//53wdHRURg9erSwZMkS4bvvvhPq6+sVfXx8fITt27crve7cuXOCvb29UFpaKgiCIEycOFEYM2aMUp+XXnpJGD16tFJbQECAsGrVqifGc+/ePcHe3l5IT08XBEEQ8vPzBXt7e+Gzzz5T9KmpqRGcnJyEHTt2KNr+9re/CXv37lUa68cffxTs7e2FwsJCQRAEIS4uTvD09BQqKiqafF8ed/78ecHe3l7Iz89Xas/NzRXs7e2FmzdvKrWvWbNGeO211xTPa2trhfHjxwvBwcHCm2++KYwbN06oqalRbF+7dq3w4osvNtivp6ensHnzZqW2yMhIYcqUKU+MtaamRhg4cKBw6NChp47z+GvmzZsn2NvbC+7u7sJbb70lbNq0Sbhz546iT2VlpWBvby8cPHhQ6XlsbKyiT0FBgWBvb6/0Oy4qKhLs7e2FU6dONTpOc+J7NEZWVpYgCP973zdt2qTU71F7ZmamIAiCkJeXJ9jb2wsZGRlK/ZKTkwUXFxdBLpcLgiAIv/32m9C3b1/h2LFjT4yBSAyuGSJqw1xdXfHdd98hMzMTly5dwvnz5zFr1ix4eXkhISEB9+7dw+3bt7Fq1Sp8+OGHitcJ///+yzdu3MCAAQMAAI6OjkpjW1hYwMLCQqnN0tISv/32m+J5dnY2PvnkE2RnZ+PevXuK9oKCAri6uiqe/3FsXV1dmJub4+7du890rFeuXIGzszOMjIye6XVP8+ispTFjxii119bWKu1HR0cHa9aswUsvvQSpVIqUlBTo6rbOf495eXlYv349Ll++jJKSEgiCgMrKymda96Wrq4sPP/wQkZGROHv2LLKysrB9+3bEx8dj8+bNcHFxeeJr//i7efT7/uPi80dreP74e2/K//3f/2HDhg346aefGvxd9OvXT/H80d/es3rxxRcRExODAwcOYOzYsdi3bx/MzMzg7e3dovGImsJkiKiN09XVhYuLC1xcXDBlyhQkJSXh3Xffxfnz5xXrRRYvXgx3d/cGr/3jlMLjH+4SiQR6enoNXlNfXw8AePjwIaZMmQJXV1esXLlS8aHp7++Pmpoapdc8Po5EIlEkZM9CIpE882ueRhAESCQS7Nmzp8HxPz79duXKFVRVVUEikaCoqAi2trbNivfx43x8qigkJAS2trZYsWIFrK2toaenh6CgoAbvYXNYW1tjzJgxGDNmDObOnYuAgABFQvQkfzzuR+/vH39fj9oe/d6bUlZWhilTpsDT0xOrVq2Cubk5ampqEBgY2OCYDA0Nm31sf6Svr49XXnkFu3fvxtixY7Fnzx6MHTu2XS2Kp7aFyRCRhunduzeA37/JDxo0CDY2Nvj111/x+uuvt+p+rl+/jpKSEsyePVuxz4sXL7YoydHT00NdXd1T+/Tt2xe7d++GXC5vtepQv379IAgCioqK4Onp+cR+d+7cweLFizFz5kwUFxcjMjIS+/fvR6dOnRTxN5YsmJubo7i4WPFcEARkZ2eje/fuAICioiLcvHkT77//PgYPHgwAyM/Px4MHD0Qfm4GBAbp164aSkhLRYz2Ln3/+Gffv30dkZCR69OgBADhz5kyLxnqUlDX2tzFu3Dhs3boVO3bsQF5eHoKCgloeNFETuICaqA2bOHEiduzYgaysLNy+fRtnzpzBihUr0KlTJ0Ul6J133sGXX36J+Ph4/Pzzz/jll19w9OhRLF26VNS+bW1toa+vjy+//BI3b97EmTNnEBMT06LqTffu3XH27FkUFRU98cP7H//4B+rr6zF9+nRcuHAB+fn5+P7775UWdD8rOzs7jB49GgsWLEBKSgpu3ryJ7Oxs7N69G5999hmA3z+I586dCwcHB0ybNg0LFiyAsbExlixZohR/YWEhsrKyUFJSgsrKSgCAp6cnkpOTcebMGVy/fh1RUVFK04NmZmbo1KkTdu3ahby8PFy4cAHz5s2DgYHBMx3Hl19+ieXLlyMtLQ03b97E9evXER8fjx9//BHDhw9v8fvTEt27d4eenh6+/PJL5OfnIy0tDbGxsS0ay8LCQnFdo7t37yoliT179oS7uztiYmIwZMgQdOvWrbUOgagBJkNEbZiXlxdSUlIQGhqKkSNHYuHChXjuueewY8cOxanjL7/8Mj7++GP88MMPeO211xAUFIT169fDyspK1L7NzMwQGxuL06dPw9/fH6tXr8b8+fNbdHbX/PnzceXKFQwbNgweHh6N9rGyssL27dthbGyM0NBQjB49usGp/i2xevVqjB8/Hp988gleeuklTJ48GcnJyfjLX/4CAEhISEBubi5iY2MhlUphYGCAuLg4/PDDD4qznUaNGgVfX19MnToVHh4e+OKLLwAAb7/9Njw9PTFz5kwEBwfDysoKPj4+in3r6enhX//6F65du4aAgAC89957CAkJgamp6TMdw8CBA1FRUYHly5dj9OjRGD9+PI4dO4alS5fi7bffFv0ePQtra2usWrUKx48fx0svvYS1a9di4cKFLRpLX18f7733HpKSkuDl5dWgujlu3DjU1NRg3LhxrRE60RNJhJbUvImIiFRs69at2LJlC06cONFqC9qJGsO/LiIialPKy8uRl5eHzz//HMHBwUyESOU4TUZERG3Ke++9hzfeeAN9+/bFP//5T3WHQ23AV199hVdffRX9+vXDggULntr3888/x5AhQ+Dq6oqFCxc2uEhsYzhNRkRERG3akSNHIJVKcfLkSVRVVWHVqlWN9jt58iTmz5+Pbdu2wcrKCjNmzMDAgQMxd+7cp47PyhARERG1aSNGjMDw4cObPPlg//79CAoKgp2dHTp37qy4nVBTmAwRERFRu3Dt2jWlq647ODjg7t27SldKbwxXpZFKrZA4NN2JRFm6p7e6Q2j3bs3MUncI7V63kO7qDkErSFe07AKZTRHzf/0y4adWi0Mul6Njx46K5yYmJgCAiooKdOnS5YmvY2WIiIiI2gUjIyOUl5crnj/62djY+KmvYzJEREREokhFPFqTnZ0dfvrpf5WmnJwcWFhYPLUq9Ch+IiIiohZTdTJUW1uLqqoq1NfXo66uDlVVVQ1uigwAgYGB2LNnD3Jzc3H//n0kJCTglVdeaVb8RERERC2m6mQoISEBAwYMwMaNG5GcnIwBAwYgISEBBQUFcHZ2RkFBAYDfb2H01ltvITg4GD4+PujWrRtmzZrV5Pi8zhCpFBdQqx4XUKseF1CrHhdQ/zlUtYB6tYj/6+e34gLqluLZZERERCSKpk8zaXr8RERERKKwMkRERESiSNQdgEhMhoiIiEgUTZ9mYjJEREREojAZIiIiIq3GZIiIiIi0GpMhIiIi0mqangxpevxEREREorAyRERERKJoemWFyRARERGJwmSIiIiItBqTISIiItJqTIaIiIhIqzEZIiIiIq2m6cmQpsdPREREJAorQ0RERCQK71pPREREWk3Tp5mYDBEREZEoTIaIiIhIqzEZIiIiIq3GZIiIiIi0mqYnQ5oePxEREZEorAwRERGRKJpeWWEyRERERKIwGSIiIiKtxmSIiIiItBqTISIiItJqmp4MaXr8pAa+vr44ffq0usMgIqI2QiLi0RYwGSIiIiKtxmRIyxUWFmLGjBkYPHgw3N3dERUVhZs3byI4OBju7u5wd3dHZGQkHjx4AACYN28eCgoKEBYWBmdnZ2zatEnNR0BEROomFfFoC9pKHKQGdXV1mDZtGmxtbXH8+HGkpqbipZdegiAImDZtGk6ePImDBw/izp07WL9+PQAgNjYWtra2SExMREZGBkJCQtR8FEREpG6angxxAbUWy8zMRHFxMd59913o6v7+pyCTyQAAzz33HADAzMwMkydPxieffKK2OImIqG1rK0lNSzEZ0mKFhYWwtbVVJEKP/Pbbb4iOjkZ6ejoqKiogCAI6deqkpiiJiKitk7SVldAtpOnJHIlgY2ODwsJC1NbWKrWvWbMGEokEycnJuHjxImJjYyEIgpqiJCKitk4qEVr8aAuYDGmxAQMGwNLSEmvWrIFcLkdVVRUuXLiAiooKGBkZoVOnTigqKsLmzZuVXmdhYYH8/Hw1RU1ERG2NRNLyR1vAZEiL6ejoIDExETdu3ICPjw+8vLxw8OBBzJgxA1evXoVMJkNoaChGjBih9LrQ0FAkJCRAJpNhy5YtaoqeiIiodUgEzn+QCq2QOKg7hHZv6Z7e6g6h3bs1M0vdIbR73UK6qzsErSBdcUYl457UtW/xa1+o/bkVI2kZLqAmIiIiUSRtZO1PSzEZIiIiIlHaytqflmIyRERERKIwGSIiIiKt1lZOkW8pJkNEREQkioYXhnhqPREREWk3VoaIiIhIFK4ZIiIiIq3GZIiIiIi0Gq8zRERERFpNysoQERERaTNOkxEREZFWk0Czp8l4aj0RERG1aaWlpQgPD4eTkxN8fHyQkpLSaL/q6mosXboUnp6eGDRoEMLCwlBUVNTk+EyGiIiISBSJpOWP5oiKioKenh5OnTqF2NhYLF++HNeuXWvQb9u2bbh06RKSk5Nx8uRJmJiY4P33329yfCZDREREJIoqkyG5XI4jR44gIiICxsbGkMlk8PX1RVJSUoO+t27dwtChQ2FhYQEDAwP4+/s3mjQ9jskQERERiSKVCC1+NCUvLw9SqRS9evVStDk6OiI3N7dB36CgIFy8eBFFRUV4+PAhUlJS4OXl1eQ+uICaiIiIRFHl2WRyuRwmJiZKbSYmJqioqGjQt1evXrC1tYWXlxd0dHRgb2+P9957r8l9sDJEREREokhEPJpiZGSE8vJypbby8nIYGxs36Lts2TJUVVXh7NmzuHTpEvz8/BASEtLkPpgMERERkSgSidDiR1N69uyJuro65OXlKdpycnLQp0+fBn1/+uknvPLKKzA1NYW+vj4mTZqEzMxMlJSUPHUfTIaIiIiozTIyMoKfnx/WrVsHuVyOCxcu4NixYwgMDGzQt3///khKSkJZWRlqamqwfft2WFlZwczM7Kn7YDJEREREoqj61Pply5ahsrISnp6eiIyMxPLly2FnZ4f09HQ4Ozsr+r377rvQ19fHiBEj4OHhgR9++AEbNmxocnwuoCYiIiJRVH1vMlNTU8THxzdol8lkyMjIUDzv0qUL1qxZ88zjMxkiIiIiUXjXeiIiItJqGn6fViZDREREJA7vWk9ERERaTdOnyXg2GREREWk1VoaIiIhIFFWfTaZqTIaIiIhIFK4ZIiIiIq3GZIiIiIi0mgSavYBaIgiCZh8BtWnC3pfUHUK7FxV0Xd0htHvLiiPVHUK7t8Lq2a8aTM9umfCTSsa91a1Hi1/b/XZ+K0bSMqwMERERkSgSDV9BzVPriYiISKuxMkRERESiSDS8tMJkiIiIiETh2WRERESk3TR8zRCTISIiIhKF02RERESk1SQaPk/GZIiIiIhE0fTKkIaHT0RERCQOK0NEREQkDqfJiIiISJtp+jQZkyEiIiISRdNvx8FkiIiIiETR8FkyJkNEREQkDqfJiIiISLtp+DSZhudyREREROKwMkRERESicM0QERERaTWeTUZERERajQuoiYiISKvxRq1ERESk3TS8MqTh4RMRERGJw8oQERERiaLhs2RMhoiIiEgcnk1GREREWo1nkxEREZF20/B5MiZDREREJAorQ0RERKTVNH3NkIbnckRERETisDJEREREomj4kiEmQ0RERCSOVk+TrV+/HnPnzm2tWOgxDg4OuHHjhrrDICIiejqJiEcb8NRkyNnZWfFwdHTEgAEDFM+Tk5P/rBifasGCBYiLi2v1cW/dugUHBwfU1ta2+tiNmTRpEnbv3v2n7EssX19fnD59Wt1hEBFRGyGRtvzRFjx1miwjI0Pxs6+vL6Kjo+Hp6aloW79+veoiIyIiIo2g1dNkAFBTU4N3330Xzs7O8Pf3R1ZWlmJbUVERZs6cicGDB8PX1xdffPHFE8eprKzEqlWr4OPjA1dXV4wfPx6VlZUAgFmzZmHIkCFwdXXFhAkTcO3aNQDArl27kJKSgi1btsDZ2RlhYWENxhUEAStXroSHhwdcXV0REBCAn3/+GQBw4sQJvPzyy3BxcYG3t7dScjdx4kQAgJubG5ydnZUSw0eqq6sRExODoUOHYujQoYiJiUF1dTUA4OzZs/Dy8sJnn30GDw8PDB06FHv37m302OPi4pCeno6oqCg4OzsjKipKse306dMYMWIE3NzcsGLFCgiCoNi2Z88ejBo1Cm5ubpg6dSpu377d6PiPqlx79+6Ft7c33NzcsGPHDmRmZiIgIAAymUxpnzdv3kRwcDDc3d3h7u6OyMhIPHjwAAAwb948FBQUICwsDM7Ozti0aVOj+yQiIu0hkbT80RaIToaOHz8Of39/pKenw9fXF++//z4AoL6+Hm+//TYcHByQmpqKbdu2Ydu2bTh58mSj46xevRpXrlzBzp07ce7cOcybNw9S6e/heXl54fDhwzhz5gyef/55xTqlcePGISAgAFOnTkVGRgYSExMbjJuWlob09HQcPnwY6enp+Pjjj2FqagoAMDQ0xOrVq5Geno5PP/0UO3bswNGjRwEAX331FQDg/PnzyMjIgLOzc4OxExIScPnyZSQlJSE5ORlZWVmIj49XbL979y7KysqQmpqKmJgYREVF4f79+w3GmT17NmQyGZYuXYqMjAwsXbpUse3EiRPYs2cPkpKScPDgQcX7d/ToUXz66af45JNPcObMGbi6uiIyMvKpv6vLly/jyJEjiIuLw8qVK5GYmIjPP/8c//nPf3Dw4EGcO3cOwO8J5LRp03Dy5EkcPHgQd+7cUSSKsbGxsLW1RWJiIjIyMhASEvLUfRIREbV1opMhV1dXeHt7Q0dHB4GBgcjJyQEAZGVloaSkBDNmzIC+vj569OiB119/HQcOHGgwRn19Pfbu3YvFixfD2toaOjo6cHFxgb6+PgAgKCgIHTt2hL6+PmbOnImcnByUlZU1Kz5dXV1UVFTgl19+gSAI6N27N6ysrAAA7u7ucHBwgFQqhaOjI/z9/RUJQXOkpKQgPDwc5ubmMDMzQ3h4uNJaKl1dXYSHh0NPTw/e3t4wMjLCr7/+2uzxASAkJASdOnWCra0t3N3dFe/vzp07ERoait69e0NXVxdhYWHIzs5+YnUIAMLDw2FgYIChQ4fCyMgIo0ePhrm5OaytrSGTyXD16lUAwHPPPYchQ4ZAX18fZmZmmDx5Ms6fP/9McRMRkfaQSCUtfrQFok+tt7CwUPzcoUMHVFVVoba2Frdv30ZxcTFkMplie11dndLzR+7du4eqqir06NGjwba6ujrExcXh0KFDKCkpUVSL7t27BxMTkybj8/DwwIQJExAVFYWCggL4+flh/vz56NixIy5fvoyPPvoI165dQ01NDaqrqzFy5MhmH3txcTFsbW0Vz21tbVFcXKx4bmpqCl3d/73FhoaGkMvlzR4fACwtLZVeX1FRAQAoKCjAypUrsXr1asV2QRBQVFSEbt26NTqWubm54mcDA4MGzx/F9ttvvyE6Ohrp6emoqKiAIAjo1KnTM8VNRERapI0shG4plV1nyMbGBt27d8eRI0ea7NulSxcYGBggPz8fjo6OSttSUlJw7NgxbN26Fd27d0dZWRnc3NwUa2ckzZhwDA4ORnBwMH777Te888472Lx5M9555x1ERkZi4sSJ2Lx5MwwMDBATE4N79+41e1wrKysUFBTAzs4OAFBYWKioOqmajY0NwsLCMGbMmFYfe82aNZBIJEhOTkaXLl1w9OhRpTVFREREStpIhaelVJbLDRgwAB07dsTGjRtRWVmJuro6/Pzzz8jMzGwYhFSKsWPH4oMPPkBRURHq6uqQkZGB6upqVFRUQF9fH126dMHDhw+xdu1apdeam5vj1q1bT4wjMzMTly9fRk1NDQwNDaGvrw8dHR0AQEVFBTp37gwDAwNkZmbi22+/VbzOzMwMUqkU+fn5Txzb398fCQkJKCkpQUlJCTZs2ICAgIBnfasA/F5he9q+HvfGG29g48aNisXkZWVlOHjwYIv2/biKigoYGRmhU6dOKCoqwubNm0XFSkRE7ZxUxKMNUFkYOjo6SEhIQE5ODoYNG4bBgwdjyZIlKC8vb7T//PnzYW9vj6CgIAwaNAgfffQR6uvr8fLLL8PW1hYvvPAC/P394eTkpPS6oKAg5ObmQiaTYfr06Q3GraiowJIlSzBo0CD4+PjA1NQUU6ZMAQAsW7YM69atg7OzMzZs2IBRo0YpXmdoaIiwsDCMHz8eMpkMly5dajD29OnT0a9fP4wZMwZjxoxB3759G42hOYKDg3H48GG4ubkhOjq6yf5+fn546623MGfOHLi4uGD06NFITU1t0b4fN2PGDFy9ehUymQyhoaEYMWKE0vbQ0FAkJCRAJpNhy5YtrbJPIiLSYFJJyx/NUFpaivDwcDg5OcHHxwcpKSlP7HvlyhVMmDABzs7O8PT0xLZt25ocXyL88VxtolYm7H1J3SG0e1FB19UdQru3rPjpZ2qSeCus1qg7BK2wTPhJJePWvdG/xa/V2ZnVZJ85c+agvr4eMTExyM7OxrRp07Bz507FMpVHSkpK4O/vj4ULF2LkyJGorq5GUVERevfu/dTx20iBioiIiKghuVyOI0eOICIiAsbGxpDJZPD19UVSUlKDvp9//jmGDh2KMWPGQF9fHx07dmwyEQKYDBEREZFYKpwmy8vLg1QqRa9evRRtjo6OyM3NbdD30qVL6Ny5M9544w14eHggLCwMBQUFTYf/bEdLRERE9BgVJkNyubzBpXRMTEwUl5r5o6KiIuzfvx+LFi3CiRMn0L17d8yZM6fp8Jt/pERERESNUOHZZEZGRg1OviovL4exsXGDvgYGBvDz88OAAQNgYGCA8PBwZGRkNHmhZiZDREREJI4KK0M9e/ZEXV0d8vLyFG05OTno06dPg74ODg5Kzx9dM7Cpc8WYDBEREZE4Kq4M+fn5Yd26dZDL5bhw4QKOHTuGwMDABn1fffVVHD16FNnZ2aipqUF8fDxcXV2bvIsCkyEiIiISR8XXGVq2bBkqKyvh6emJyMhILF++HHZ2dkhPT1e6kbqHhwdmz56N0NByEsOPAAAgAElEQVRQeHp64ubNm1izpunLNqjsdhxERERErcHU1BTx8fEN2mUyGTIyMpTa/vGPf+Af//jHM43PZIiIiIjE0exbkzEZIiIiIpE0/EatTIaIiIhIHCZDREREpNU0/HQsJkNEREQkDitDREREpM0kGl4Z0vDwiYiIiMRhZYiIiIjE4TQZERERaTUNn2diMkRERETisDJEREREWo3JEBEREWk1TpMRERGRVtPwypCG53JERERE4rAyREREROJoeGmFyRARERGJo+HTZEyGiIiISBxWhoiIiEirsTJEREREWo2VISIiItJqGl4Z0vBcjoiIiEgcVoaIiIhIHA2vDDEZIiIiInE0fJ6JyRARERGJw8oQ0ZPdmpml7hDavWXF76k7hHZvhdUadYfQ7i3dZ6fuEEgMVoaIiIhIq0lYGSIiIiJtptm5kKYXtoiIiIjEYWWIiIiIxOE0GREREWk1zc6FmAwRERGRSKwMERERkVbT8BXITIaIiIhIHFaGiIiISKtpdi6k6YUtIiIiInFYGSIiIiJxOE1GREREWk2zcyEmQ0RERCQSK0NERESk1TR8BTKTISIiIhKHlSEiIiLSapqdC2l6YYuIiIhIHFaGiIiISBxOkxEREZE20/BciMkQERERiaTh2RCTISIiIhJHs3MhJkNEREQkklSzsyEmQ0RERCSOZudCPLWeiIiItBsrQ0RERCSOhi+gZmWIiIiIxJGIeDRDaWkpwsPD4eTkBB8fH6SkpDy1f3V1NUaOHAkvL69mjc/KEBEREYmj4spQVFQU9PT0cOrUKWRnZ2PatGlwdHSEnZ1do/23bNkCc3NzyOXyZo3PyhARERGJo8LKkFwux5EjRxAREQFjY2PIZDL4+voiKSmp0f75+flITk5GaGhos8NnMkRERETiSCUtfzQhLy8PUqkUvXr1UrQ5OjoiNze30f7R0dGYM2cOOnTo0Pzwm92TiIiIqDEqrgyZmJgotZmYmKCioqJB3++++w61tbXw8/N7pvC5ZoiIiIjaLCMjI5SXlyu1lZeXw9jYWKlNLpcjNjYWGzdufOZ9MBnScgsWLIC1tTVmz56t7lCIiEhTqXABdc+ePVFXV4e8vDz07NkTAJCTk4M+ffoo9btx4wZu376NCRMmAABqampQVlaGIUOGYNeuXejevfsT98FkiIiIiMRR4clkRkZG8PPzw7p16xAdHY3s7GwcO3YMO3fuVOpnZ2eHEydOKJ5nZGQgKioK+/btg5mZ2VP3wTVDREREJI5E0vJHMyxbtgyVlZXw9PREZGQkli9fDjs7O6Snp8PZ2RkAoKurC0tLS8Wjc+fOkEqlsLS0hI6OzlPHZ2VIy1y9ehWLFy9GXl4evL29IfnDH+LXX3+NTZs24f79+3BxccGKFStgbW0NAEhLS8P777+Pu3fvIiAgALm5uQgMDMRrr72mrkMhIqK2QsUXoDY1NUV8fHyDdplMhoyMjEZf4+7ujtTU1GaNz8qQFqmurkZ4eDgCAwNx7tw5jBw5EkeOHAEAnDlzBmvWrMHHH3+MtLQ0dOvWDXPmzAEAlJSUYNasWYiMjMTZs2fRq1evJ/7xERGRFlLhqfV/BiZDWuTy5cuoqanBm2++CT09PYwcORL9+/cHAKSkpGDs2LHo27cv9PX1MWfOHFy6dAm3bt1Camoq7OzsMGLECOjq6iI4OBgWFhZqPhoiImozVDxNpmpMhrRIcXExrK2tlabGbG1tFdu6deumaDc2NoapqSmKiopQXFyMrl27KrZJJBKl50RERJqMyZAWsbS0RFFREQRBULQVFBQAAKysrHD79m1Fu1wuR2lpKaytrRWve0QQBNy5c+fPC5yIiNo2VoZIUzg5OUFXVxdffPEFamtrceTIEWRlZQEAAgIC8M033yA7OxvV1dVYu3YtBgwYgO7du8Pb2xs//fQTjh49itraWvz73//G3bt31Xw0RETUZjAZIk2hr6+P9evXY9++fXBzc8OBAwcUlyz38PBAREQEZs6ciaFDhyI/Px9xcXEAADMzM/zrX/9CbGws3N3dkZubi379+kFPT0+dh0NERG2FRNryRxvAU+u1TP/+/bF///5Gt40fPx7jx49vdJuXlxe8vLwAAPX19fDy8uK6ISIi+l0bOSuspdpGSkZt3smTJ/HgwQNUV1cjMTERwO/TbkRERJo+TcbKEDXLpUuXMHfuXFRXV6NPnz7YsGEDOnTooO6wiIioLWgj010txWSImmXmzJmYOXOmusMgIiJqdUyGiIiISJw2Mt3VUkyGiIiISBwNX0DNZIiIiIjE4ZohIiIi0mqcJiMiIiKtxmSIiIiItJqGT5NpdvREREREIrEyREREROLwbDIiIiLSalwzRERERFpNw9cMMRkiIiIicVgZIiIiIq3GNUNERESk1TR8mkyzoyciIiISiZUhIiIiEodrhoiIiEirMRkiIiIircZkiIiIiLSaVLOXIDMZIiIiInFYGSIiIiKtpuHJkGbXtYiIiIhEYmWIiIiIxNHwiy4yGSIiIiJxeDsOIiIi0moavmaIyRARERGJw2kyIiIi0mqsDBEREZFW0/BkSLPrWkREREQisTJERERE4vB2HERP1i2ku7pDaPdWWK1Rdwjt3tJ9duoOod2LeuWaukPQCssEVY2s2dNkTIaIiIhIHA1fM8RkiIiIiMThqfVERESk3VgZIiIiIm2m4dNkml3XIiIiIhKJlSEiIiISh2uGiIiISLtp9jQZkyEiIiISR8PXDDEZIiIiIpE0e5pMs6MnIiIi9ZNIWv5ohtLSUoSHh8PJyQk+Pj5ISUlptN/mzZsxevRoODs7w9fXF5s3b27W+KwMERERkTgqniaLioqCnp4eTp06hezsbEybNg2Ojo6ws1O+VY4gCFi9ejUcHBxw8+ZNTJ06FTY2NvD393/q+KwMERERUZsll8tx5MgRREREwNjYGDKZDL6+vkhKSmrQNyQkBH379oWuri7++te/YtiwYbh48WKT+2AyRERERCJJRDyeLi8vD1KpFL169VK0OTo6Ijc396mvEwQB6enp6NOnT5P74DQZERERiaPC6wzJ5XKYmJgotZmYmKCiouKpr1u/fj3q6+sxduzYJvfBZIiIiIjEUeGaISMjI5SXlyu1lZeXw9jY+Imv+eqrr7B//35s374d+vr6Te6D02REREQkkuqmyXr27Im6ujrk5eUp2nJycp44/bVnzx5s3LgR27ZtQ9euXZsVPZMhIiIiEkcibfmjCUZGRvDz88O6desgl8tx4cIFHDt2DIGBgQ36JicnIy4uDlu3bkWPHj2aHT6TISIiIhJFIpG0+NEcy5YtQ2VlJTw9PREZGYnly5fDzs4O6enpcHZ2VvT7+OOPUVpaiqCgIDg7O8PZ2RlLly5tcnyuGSIiIqI2zdTUFPHx8Q3aZTIZMjIyFM+PHz/eovGZDBEREZFIvDcZERERaTMVnlr/Z2AyRERERCKxMkRERETaTMX3JlM1JkNEREQkDqfJiIiISLtpdmVIs1M5IiIiIpFYGSIiIiJxuGaIiIiItBrXDBEREZF2Y2WIiIiItBmnyYiIiEi7afY0WatEv379esydO7c1hqJGODg44MaNGyoZ29fXF6dPn1bJ2ERERJqgWZUhZ2dnxc8PHz6Evr4+dHR0AAArVqxQTWTPaMGCBbC2tsbs2bNbddxbt25h2LBhuHLlCnR1VV9ImzRpEsaMGYPXXntN5fsiIiJqFdowTZaRkaH42dfXF9HR0fD09FS0rV+/vvUjIyIiIs2g4clQq03y1dTU4N1334WzszP8/f2RlZWl2FZUVISZM2di8ODB8PX1xRdffPHEcSorK7Fq1Sr4+PjA1dUV48ePR2VlJQBg1qxZGDJkCFxdXTFhwgRcu3YNALBr1y6kpKRgy5YtcHZ2RlhYWINxBUHAypUr4eHhAVdXVwQEBODnn38GAJw4cQIvv/wyXFxc4O3trZTcTZw4EQDg5uYGZ2dnpcTwkerqasTExGDo0KEYOnQoYmJiUF1dDQA4e/YsvLy88Nlnn8HDwwNDhw7F3r17Gz32uLg4pKenIyoqCs7OzoiKilJsO336NEaMGAE3NzesWLECgiAotu3ZswejRo2Cm5sbpk6ditu3bz/x/d2/fz98fHzg7u6OhISEZh8HAGzatEmxbffu3SqdviMiIk0iFfFQv1aL4vjx4/D390d6ejp8fX3x/vvvAwDq6+vx9ttvw8HBAampqdi2bRu2bduGkydPNjrO6tWrceXKFezcuRPnzp3DvHnzIJX+HqaXlxcOHz6MM2fO4Pnnn1esUxo3bhwCAgIwdepUZGRkIDExscG4aWlpSE9Px+HDh5Geno6PP/4YpqamAABDQ0OsXr0a6enp+PTTT7Fjxw4cPXoUAPDVV18BAM6fP4+MjAylKcNHEhIScPnyZSQlJSE5ORlZWVmIj49XbL979y7KysqQmpqKmJgYREVF4f79+w3GmT17NmQyGZYuXYqMjAwsXbpUse3EiRPYs2cPkpKScPDgQcX7d/ToUXz66af45JNPcObMGbi6uiIyMrLR9zY3NxcrVqzAhx9+iJMnT6K0tBR37txp1nGkpqbi888/x9atW/Hdd9/h3Llzje6DiIi0kETS8kcb0GrJkKurK7y9vaGjo4PAwEDk5OQAALKyslBSUoIZM2ZAX18fPXr0wOuvv44DBw40GKO+vh579+7F4sWLYW1tDR0dHbi4uEBfXx8AEBQUhI4dO0JfXx8zZ85ETk4OysrKmhWfrq4uKioq8Msvv0AQBPTu3RtWVlYAAHd3dzg4OEAqlcLR0RH+/v7P9GGfkpKC8PBwmJubw8zMDOHh4UhOTlbad3h4OPT09ODt7Q0jIyP8+uuvzR4fAEJCQtCpUyfY2trC3d1d8f7u3LkToaGh6N27N3R1dREWFobs7OxGq0OHDh3C3//+d7i5uUFfXx8RERGKRLOp4zh48CBeffVV2NnZwdDQEDNmzHim+ImIqD2TiHioX6utCLawsFD83KFDB1RVVaG2tha3b99GcXExZDKZYntdXZ3S80fu3buHqqoq9OjRo8G2uro6xMXF4dChQygpKVF8iN+7dw8mJiZNxufh4YEJEyYgKioKBQUF8PPzw/z589GxY0dcvnwZH330Ea5du4aamhpUV1dj5MiRzT724uJi2NraKp7b2tqiuLhY8dzU1FRp8bWhoSHkcnmzxwcAS0tLpddXVFQAAAoKCrBy5UqsXr1asV0QBBQVFaFbt24N4uzataviuZGRkaI61tRxFBcXo1+/foptNjY2zxQ/ERG1Y7wC9dPZ2Nige/fuOHLkSJN9u3TpAgMDA+Tn58PR0VFpW0pKCo4dO4atW7eie/fuKCsrg5ubm2LtjKQZpbbg4GAEBwfjt99+wzvvvIPNmzfjnXfeQWRkJCZOnIjNmzfDwMAAMTExuHfvXrPHtbKyQkFBAezs7AAAhYWFiqqTqtnY2CAsLAxjxoxpsq+VlRWuX7+ueP7w4UOUlpYqbX/ScVhZWaGoqEjRt7CwsLUOgYiISK1UnsoNGDAAHTt2xMaNG1FZWYm6ujr8/PPPyMzMbBiMVIqxY8figw8+QFFREerq6pCRkYHq6mpUVFRAX18fXbp0wcOHD7F27Vql15qbm+PWrVtPjCMzMxOXL19GTU0NDA0NlS4PUFFRgc6dO8PAwACZmZn49ttvFa8zMzODVCpFfn7+E8f29/dHQkICSkpKUFJSgg0bNiAgIOBZ3yoAv1fYnravx73xxhvYuHGjYjF5WVkZDh482GjfF198ESdOnEB6ejqqq6uxbt061NfXN+s4Ro4ciW+++QbXr1/Hw4cPsWHDhhYdHxERtUNcM/R0Ojo6SEhIQE5ODoYNG4bBgwdjyZIlKC8vb7T//PnzYW9vj6CgIAwaNAgfffQR6uvr8fLLL8PW1hYvvPAC/P394eTkpPS6oKAg5ObmQiaTYfr06Q3GraiowJIlSzBo0CD4+PjA1NQUU6ZMAQAsW7YM69atg7OzMzZs2IBRo0YpXmdoaIiwsDCMHz8eMpkMly5dajD29OnT0a9fP4wZMwZjxoxB3759G42hOYKDg3H48GG4ubkhOjq6yf5+fn546623MGfOHLi4uGD06NFITU1ttK+dnR2WLl2KuXPn4oUXXkCnTp2Ups2edhze3t6YNGkSgoOD4efnp3j/H63nIiIibabZa4Ykwh/P0SZqpuvXr2P06NHIysp66sUo65d5/IlRaaf3o0rUHUK7t3SfnbpDaPeiXrmm7hC0wjLhJ9UMfH9ny1/b+Y3Wi6OFNHvFE/2pvvvuO1RXV+P+/fuIjY2Fj4/Pn3JVbiIiaus0uzLEZIiabefOnfDw8ICfnx90dHSwfPlydYdERERtgYavGeLXemq2LVu2qDsEIiJqkzS7tqLZ0RMRERGJxMoQERERidNGprtaiskQERERicMrUBMREZF2Y2WIiIiItBmnyYiIiEi7cZqMiIiItJmGV4Y0O5UjIiIiEomVISIiIhJJs2srTIaIiIhIHA2fJmMyREREROIwGSIiIiLtxmkyIiIi0masDBEREZF20+xkSLPrWkREREQisTJERERE4vBGrURERKTdNHuajMkQERERicPKEBEREWk3VoaIiIhIm/HUeiIiItJqGj5NptnRExEREYnEyhARERGJxGkyIiIi0mYavmaI02REREQkklTEo2mlpaUIDw+Hk5MTfHx8kJKS0mg/QRAQGxsLd3d3uLu748MPP4QgCE2Oz8oQERERiaPiylBUVBT09PRw6tQpZGdnY9q0aXB0dISdnZ1Sv127duHo0aNISkqCRCLB5MmT0aNHD4wfP/6p47MyRERERCKprjIkl8tx5MgRREREwNjYGDKZDL6+vkhKSmrQd//+/ZgyZQq6du0Ka2trTJ48Gfv27WtW9EREREQtJ5G0/NGEvLw8SKVS9OrVS9Hm6OiI3NzcBn2vXbsGR0dHpX7Xrl1rch+cJiOVkq44o+4Q2r1lK9QdAZF4y5pe1kFtmqvKRpbL5TAxMVFqMzExQUVFRaN9O3bsqNRPLpdDEARInpJ4sTJEREREbZaRkRHKy8uV2srLy2FsbNxo3z8mSeXl5TAyMnpqIgQwGSIiIqI2rGfPnqirq0NeXp6iLScnB3369GnQ187ODjk5OUr9Hl9k3RgmQ0RERNRmGRkZwc/PD+vWrYNcLseFCxdw7NgxBAYGNugbGBiIrVu3oqioCEVFRdi6dSteeeWVJvchEZpzAj4RERGRmpSWlmLRokU4ffo0TE1NERkZiYCAAKSnpyMkJAQZGRkA/nedoT179gAAgoKCMG/evCanyZgMERERkVbjNBkRERFpNSZDREREpNWYDBGRSm3ZsqXR9q1bt/7JkRCJk5+f3+ijqKgI9fX16g6PROCaIdJ6W7ZswdSpUxu0b926FZMnT1ZDRO2Li4sLLl682KB90KBBOHfunBoiImoZR0dHxULcxy/iJ5VK4evri2XLlsHCwkJdIVILMRkirccPa9U4c+b3q4+//fbbSExMVLpz9K1btxAfH4/vv/9eXeG1O4/Onnmcvr4+unbtCicnJ+jr6//JUbUvu3fvxvnz5zFjxgx07doVhYWFiI+Ph7OzM9zc3PDRRx9BT08P69atU3eo9IyYDJHW4oe1avn6+gIACgsLYWNjo2iXSCSwtLRESEgIhg0bpq7w2p1JkyYhIyMDFhYW6Nq1K+7cuYO7d++iX79+uH37NgAgPj4e/fv3V3OkmsvLywvfffcdDAwMFG0PHz7Eiy++iNTUVNy/fx8jRozA2bNn1RgltQTvTUZaa/HixQCAqqoqLFq0SNH+6MN6yZIl6gqtXTh+/DgA4N1338WHH36o5mjavz59+sDPzw/BwcGKtq+++gq//PILduzYgYSEBERHR2PXrl1qjFKz1dfX49atW+jdu7eiraCgQLFeyMjICHV1deoKj0RgZYi0Hj+sqT1wc3PD2bNnIZX+77yYuro6DB48GOfPn0d1dTU8PDxw4cIFNUap2TZt2oRt27Zh7NixiurbN998g0mTJiE0NBRHjx7Fzp07sXnzZnWHSs+IlSHSekyEVKu8vBzr16/H+fPnce/ePaXpyBMnTqgvsHbG3Nwcx48fx/DhwxVtJ06cgJmZGYDfK6C6uvwvX4yQkBA4ODjg0KFDuHLlCiwtLRETEwMvLy8AwPDhw5Xef9Ic/JdBWo8f1qq1fPlyFBUVYfr06Zg3bx5iY2OxZcsWvPjii+oOrV1ZsmQJIiIiYGdnBxsbGxQWFuLatWv417/+BQC4fPkyJk2apOYoNZ+Xl5ci+aH2g9NkpPXmzp2LoqIivPnmmw0+rP/5z3+qOzyN5+HhgQMHDqBLly6QyWRIT09HUVERwsLCsG/fPnWH166UlJQgNTUVxcXFsLKygre3N7p06aLusNqNmpoaJCQkICkpSfEeBwYGIiwsjGfqaThWhkjrnTp1SvFhraOjg+HDh6N///4ICwtjMtQK6uvrYWJiAuD3BaYPHjyApaUlbty4oebI2h8zMzMMGjQIRUVFsLa2ZiLUymJjY5GZmYkVK1bA1tYWBQUFiI+PR3l5udJJGKR5mAyR1uOHtWo5Ojri/Pnz8PDwgEwmw4oVK2BsbIyePXuqO7R2pbi4GHPmzMGlS5dgamqK0tJSDBw4EGvXroW1tbW6w2sXDh06hKSkJEWS+de//hXPP/88AgMDmQxpON6Og7Teow9rAIoP6+XLl/PDupVER0ejW7duAH5f19KhQwc8ePCAC9db2fLly+Ho6Ihz584hLS0N586dw9/+9jcsW7ZM3aG1G09aVcLVJpqPa4ZI6+Xn5wMAevTogZKSEqxZswZyuRwzZsxQup4IUVvm7u6OtLQ06OnpKdqqq6vxwgsv8CKArSQmJgZZWVkIDw+Hra0tbt++jYSEBPTr109x3TLSTJwmI623bds2vPTSS+jRowfMzMwQExODixcvYufOnfwPrpWkp6fj6tWrkMvlSu1hYWFqiqj96dy5M65fvw5HR0dF2y+//IJOnTqpMar2Zd68eUhISEBUVJRiAbW/vz+mT5+u7tBIJFaGSOsNHjwYqampSmeDVFdXw9vbW3HLDmq5999/HwcPHoRMJlO6jYFEIuFUWSv6+uuvsXbtWgQFBSmqFvv27UNERATGjRun7vCI2jRWhkjrSSSSBnP+dXV1ikvskzgpKSlISUnhIl4Ve/311/GXv/wFKSkp+Omnn2BtbY01a9bAw8ND3aFptOZ+IeL7rNmYDJHWk8lk+PjjjzFv3jxIpVLU19dj/fr1kMlk6g6tXejatSuvwaIijy6o+EdWVlawtLSERCLBuXPncO7cOURERKghuvbh8any4uJiAFCcsQcA1tbWOHbs2J8eG7UeJkOk9RYvXoxp06Zh6NChsLW1RWFhISwtLZGYmKju0NqFmJgYvPfee/D394eFhYXSNjc3NzVF1T7cuXNH8XNVVRWOHDmCfv36oVu3bigoKEBWVhZGjBihxgg136MbDgNAYmIiSktLERERAUNDQzx8+BDr1q2DqampGiOk1sA1Q0T4/VpDmZmZKCwshI2NDQYMGKB0w0tquZ07d2LlypUwNDREhw4dFO0SiYS3O2lFs2fPxsiRI5Vuc3LkyBEcOnQIa9euVWNk7cfgwYNx8uRJpTP2ampq8MILL+DHH39UY2QkFitDRACkUimcnJzg5OSk7lDanbi4OCQmJsLT01PdobRrqamp+Oijj5Tahg0bhoULF6opovbHyMgImZmZcHV1VbRlZWXB0NBQjVFRa2AyREQqZWhoyPVXf4LnnnsO//73vxEcHKxo2759O/7yl7+oMar2ZdasWXjrrbfg6+uLrl274s6dO/j++++xdOlSdYdGInGajIhU6ptvvkFmZibCw8Nhbm6utI1Tka3n6tWrmDFjBmpra2FtbY2ioiLo6upi/fr16Nu3r7rDazdyc3Nx+PBhFBcXw9LSEiNHjkSfPn3UHRaJxGSIiFTq0UUAJRKJok0QBEgkEmRnZ6srrHappqYGly9fVnxQOzk5Ka1vIaLGMRkiIpW6ffv2E7c9umcZEZE6MRkiIiIircYJeyIiItJqTIaIiIieQWFhIS5duqTuMKgVMRkiIiJqhoKCArzxxhsYNWoUJk+eDAA4dOhQg1t2kOZhMkREfwp+myZNt3TpUvz973/HxYsXoav7+2X6hgwZgtOnT6s5MhKLyRARqRS/TVN7kZWVhdDQUEilUsWlIkxMTFBWVqbmyEgsJkNEpFL8Nk3thbm5OW7cuKHUlpubCxsbGzVFRK2FyRARqRS/TVN7MWXKFISFhWHv3r2ora3Ft99+i9mzZyMkJETdoZFIvDcZEanUo2/TvXr1UrTx2zRpoqCgIJiammLXrl2wsbHBvn37EBERgeHDh6s7NBKJyRARqdSjb9OhoaGKb9Offvopv02TRho+fDiTn3aIV6AmIpU7evQodu3ahYKCAnTt2hXjx4/nBwpppLS0NGRnZ0Mulyu1R0REqCkiag1MhoiIiJohKioKBw8ehLu7OwwNDZW2ffDBB2qKiloDkyEiUjl+m6b2wN3dHfv37+d6t3aIa4aISKWe9m2aSJOYmprCxMRE3WGQCrAyREQqxW/TpMny8/MVP586dQonTpzAtGnTYGFhodSvR48ef3Zo1IpYGSIileK3adJkfn5+kEgk+GPd4MSJE0p9JBIJsrOz/+TIqDWxMkRErY7fpolIkzAZIqJW5+jo2ODb9OP4bZo0TXR0NJYsWdKgPSYmhvfa03BMhoiIiJrBxcUFFy9ebNDu7u6Os2fPqiEiai1cM0REKsVv06Tp9uzZAwCoq6tT/PxIfn4+TE1N1REWtSJWhohIpfhtmjTdpEmTAAAXLlyAq6urol0ikcDCwgLBwcFwcnJSV3jUClgZIiKV4Ldpai++/PJLAEBcXBxmz+pjZGwAAAukSURBVJ6t5mhIFVgZIiKV4LdpItIUTIaISKX4bZqI2jomQ0RERKTVpOoOgIiIiEiduICaiIiomcrKyvDrr7+ioqJCqd3Dw0NNEVFrYDJERETUDN988w2ioqJgZGSEDh06KNolEgmOHTumxshILK4ZIiKV47dpag9eeOEFREdHw9vbW92hUCtjZYiIVIrfpqm9qKurw9ChQ9UdBqkAK0NEpFL8Nk3txdatW1FRUYHp06dDKuX5R+0JkyEiUilPT0+cPHkSOjo66g6FSBRvb2/cvXsXenp6Da6gfuLECfUERa2CyRARqRS/TVN7ce7cuSduGzRo0J8YCbU2JkNEpFL8Nk1EbR2TISJSKX6bpvaipqYGCQkJSEpKQnFxMaysrBAYGIiwsDDo6+urOzwSgckQERFRM6xcuRKZmZmYMWMGbG1tUVBQgPj4ePTr1w+LFi1Sd3gkApMhIlIpfpum9sLLywtJSUno0qWLoq2kpASBgYH4f+3dfWiV5R/H8c+ZnT2ILif2sAcrlhsRNdt0rUERQ1KKyaIw/7ARlsWWCtVxmVOmCdVSJAJpM6gog0BRghYJNkWaPeyY2hnOaYlhTUqkDTnH2HGd+/eH7ND5abF1n2uX9733Cw6cXWd/fP4Y7Pu5znXf91dffWUxGdziPkMAjNq8ebMikYheffXVlDYdjUZp0/CUf9o7YE/B+9gZAmAUbRp+8dprr6mnp0fLly9XQUGB+vv71dbWprvuuktr1661HQ8usDMEwCjaNPyiqalJbW1t2rhxo86dO6ebbrpJjzzyiJ5//nnb0eASO0MAjKJNA7jWMQwBMCoej6utrU0dHR1XtGkOUAO4FjAMAQCACY174wMAgAmNYQgAgFH44YcfrroeiUTGOQnSjWEIAIBRWLp06VXXly1bNs5JkG4MQwCMok3D6xKJhP766y85jiPHcZRIJJKvn3/+WZMmTbIdES5xgBqAURUVFTp8+PAV6/fee++/PsQVuFbccccdCgQCV/0sIyNDDQ0NWrly5TinQjpx00UARiQSiWSTHnmNOHPmDG0antHZ2SnHcVRfX6+PP/44uR4IBDR9+nRlZ2dbTId0YGcIgBG0aQBewTAEwIj+/n7aNHyns7NT4XBYAwMDKbudmzZtspgKbnGAGoARhYWFKioq0v79+1VYWJh8FRQUMAjBk7Zu3ar169crkUhoz549mjZtmrq6upSbm2s7GlxiZwiAcbRp+EFNTY22bdum0tJSzZ07V4cOHVIkEtE777yj9vZ22/HgAjtDAIyiTcMvLly4oNLSUklSMBjUpUuXVFZWpnA4bDkZ3GIYAmDUrl279P7776u5uVnBYFDNzc1qb2/Xr7/+ajsaMCa33HKLfvzxR0lSSUmJPvnkE3366ae6/vrrLSeDW1xaD8Ao2jT84oUXXtDg4KAkadWqVQqFQrp48aLWr19vORncYhgCYNRImy4pKUm26dzcXNo0POfBBx9Mvi8rK9PevXstpkE6cYAagFEHDhzQ5MmTVVlZqUgkktKm58+fbzseMGo//fSTpk2bphkzZigWi+m9997TpEmT9PTTTysnJ8d2PLjAMAQAwCjU1dXprbfeUnFxsVpaWnT69GllZWUpLy9Pmzdvth0PLvA1GQCjaNPwi/7+fhUXF8txHH355Zfq6OhQdna25s2bZzsaXOJqMgBGhUIhXbhwQZL05ptvKhwO68iRI2ppabGcDBibzMxMRaNRRSIR3XzzzZo+fboyMzM1NDRkOxpcYmcIgFG0afhFbW2tnnrqKcViMT355JOSpN7eXhUVFVlOBrcYhgAYNdKmT506lWzTw8PDtGl4TnNzs7q6unTdddfpvvvuk3T5WXtr1qyxnAxuMQwBMIo2DT+5//77U36+++67LSVBOnE1GQDj/r9N9/T0KBqNqrq62nIyAGAYAgAAExxXkwEAgAmNYQgAgFFobGy86vqKFSvGOQnSjWEIAIBR+O6776663t3dPc5JkG5cTQbAqMbGRrW1tV2xvmLFCm3dutVCImBs3n77bUnSpUuXku9H/PLLLyooKLARC2nEMATAKNo0vO63336TJDmOk3w/Ij8/XytXrrQRC2nEMATACNo0/OKNN96QJJWXl+uJJ56wnAYmMAwBMII2Db8ZGYSi0agGBgZSPps5c6aNSEgT7jMEwKgdO3bQpuELp06dUigUUl9fnwKBgBzHUSAQkCQdP37ccjq4wTAEYFzQpuF19fX1uvPOO7V8+XLNmzdP+/bt05YtW1ReXq66ujrb8eACwxAAo2jT8IvKykp9/fXXCgaDmjt3rg4dOqSLFy+qtrZW+/btsx0PLnCfIQBGbdiwQVVVVeru7taUKVMUDoe1ePFitba22o4GjElWVpaGh4clSXl5eTp79qwSiYQGBwctJ4NbDEMAjOrr69OqVauUm5srx3E0depUvfzyy1dcYQZc6+bMmaMvvvhCkrRgwQI9++yzqq+vTz6AGN7F1WQAjBpp08FgMNmmc3NzadPwnL8P8C+99JJKSkoUi8X06KOPWkyFdGAYAmDUSJt+7LHHkm06MzOTNg3PicfjCgQCCgaDysjIUF1dneLxuO1YSAMOUAMYN4lEQp999lmyTU+ePNl2JGDUlixZoqamJt1zzz3JtaNHj2rLli3avn27xWRwi2EIgFF/b9N/X5OkzMxMW7GAMausrFR3d3fyakjp8oBfVVWlcDhsMRnc4gA1AKOWLl2qY8eOpaz19vbqmWeesZQI+G+mTp2q8+fPp6ydP39eOTk5lhIhXRiGABh18uRJzZ49O2WtrKxMfX19lhIB/838+fMVCoV08uRJ/fnnnzpx4oRWr16thx9+2HY0uMQwBMAo2jT84sUXX9Ttt9+uRYsWqaKiQosXL1ZxcbFCoZDtaHCJM0MAjGptbVVvb6/WrVunmTNn6syZM2ptbVVpaanWrFljOx4wZo7jaGBgQHl5eSnnh+BdDEMAjBoaGlJra6t2796teDyurKwsPf7441q9ejUHqAFcExiGAIwL2jSAaxXDEAAAmNA4QA0AACY0HscBAMAoHTx4UJ9//rn++OMPtbe3q6enR9FoVNXV1bajwQV2hgAAGIXt27drw4YNuu2225J3nM7Ozk55gCu8iWEIgHEHDx5Uc3OzGhoaJEk9PT365ptvLKcCxubDDz/UBx98oOeee04ZGZf/fRYXF+v06dOWk8EthiEARtGm4RexWEz5+fmSlLwicnh4OOW5e/AmhiEARtGm4ReVlZV69913U9Y++ugjVVVVWUqEdOEANQCjaNPwi3Xr1qmhoUE7d+5ULBbTggULNGXKFLW3t9uOBpcYhgAYNdKmGxsbk2u0aXjRjTfeqF27dikSiejs2bPKz89XWVlZcscT3sVNFwEYde7cOTU0NGhwcFC///67ioqKkm36hhtusB0P+E8SiUTKzwxE3sYwBMA4x3Fo0/C8Y8eOaePGjTpx4oSGhoYkXf7bDgQCOn78uOV0cINhCMC4oU3DyxYuXKiamhrV1dUpOzs75bPCwkJLqZAODEMAjKJNwy8qKir0/fff86BhH+IANQCjXnnlFdXU1Oj111+/ok0DXvLQQw+pq6tLDzzwgO0oSDN2hgAYRZuGlzU1NSX/duPxuPbv3685c+ZoxowZKb+3adMmG/GQJuwMATCKNg0vu/XWW1N+njVrlqUkMImdIQBpR5uGn3R0dKi2ttZ2DBjEzhCAtKNNw09aWloYhnyOnSEARtCm4Rfl5eU6cuSI7RgwiGEIgBEVFRU6fPiw7RiAa7Nnz9a2bdv0b/8uq6urxzER0o2vyQAYQc+CX8Tjca1du/Yf/6YDgYA6OzvHORXSiWEIgBGJRELffvstbRqel5OTw7DjcwxDAIygTQPwCoYhAEbQpuEXfOXrfzwlEQCAf8GVZP7HMATACNo0AK/g0noAADChsTMEAAAmNIYhAAAwoTEMAQCACY1hCAAATGgMQwAAYEL7H8JxIgcugSHNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f7ce409d6d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "messages = [\"cat\", \"The cat sat on the mat\", \"dog\", \"The cat sat on the dog\"]\n",
    "my_embeddings = create_embeddings(messages, embed)\n",
    "plot_similarity(messages, my_embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which is cat more similar to, \"The cat sat on the mat\" or \"dog\"? Is this desireable?\n",
    "\n",
    "Think back to how an RNN scans a sequence and maintains its state. Naive methods of embedding composition (mapping many to one) can't possibly compete with a network trained for this very purpose!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Task 4: Assessing the Embeddings Formally</h2>\n",
    "Of course, it's great to know that our embeddings match our intuitions to an extent, but it'd be better to have a formal, data-driven measure of the quality of the representation.\n",
    "\n",
    "Researchers have\n",
    "The STS Benchmark http://ixa2.si.ehu.es/stswiki/index.php/STSbenchmark provides an intristic evaluation of the degree to which similarity scores computed using sentence embeddings align with human judgements. The benchmark requires systems to return similarity scores for a diverse selection of sentence pairs. Pearson correlation is then used to evaluate the quality of the machine similarity scores against human judgements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz\n",
      "417792/409630 [==============================] - 1s 3us/step\n"
     ]
    }
   ],
   "source": [
    "def load_sts_dataset(filename):\n",
    "  # Loads a subset of the STS dataset into a DataFrame. In particular both\n",
    "  # sentences and their human rated similarity score.\n",
    "  sent_pairs = []\n",
    "  with tf.gfile.GFile(filename, \"r\") as f:\n",
    "    for line in f:\n",
    "      ts = line.strip().split(\"\\t\")\n",
    "      # (sent_1, sent_2, similarity_score)\n",
    "      sent_pairs.append((ts[5], ts[6], float(ts[4])))\n",
    "  return pd.DataFrame(sent_pairs, columns=[\"sent_1\", \"sent_2\", \"sim\"])\n",
    "\n",
    "\n",
    "def download_and_load_sts_data():\n",
    "  sts_dataset = tf.keras.utils.get_file(\n",
    "      fname=\"Stsbenchmark.tar.gz\",\n",
    "      origin=\"http://ixa2.si.ehu.es/stswiki/images/4/48/Stsbenchmark.tar.gz\",\n",
    "      extract=True)\n",
    "\n",
    "  sts_dev = load_sts_dataset(\n",
    "      os.path.join(os.path.dirname(sts_dataset), \"stsbenchmark\", \"sts-dev.csv\"))\n",
    "  sts_test = load_sts_dataset(\n",
    "      os.path.join(\n",
    "          os.path.dirname(sts_dataset), \"stsbenchmark\", \"sts-test.csv\"))\n",
    "\n",
    "  return sts_dev, sts_test\n",
    "\n",
    "\n",
    "sts_dev, sts_test = download_and_load_sts_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the data. The sim score is higher when the sentences are more similar and lower when they are not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sent_1</th>\n",
       "      <th>sent_2</th>\n",
       "      <th>sim</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A man with a hard hat is dancing.</td>\n",
       "      <td>A man wearing a hard hat is dancing.</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A young child is riding a horse.</td>\n",
       "      <td>A child is riding a horse.</td>\n",
       "      <td>4.75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A man is feeding a mouse to a snake.</td>\n",
       "      <td>The man is feeding a mouse to the snake.</td>\n",
       "      <td>5.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A woman is playing the guitar.</td>\n",
       "      <td>A man is playing guitar.</td>\n",
       "      <td>2.40</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A woman is playing the flute.</td>\n",
       "      <td>A man is playing a flute.</td>\n",
       "      <td>2.75</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 sent_1  \\\n",
       "0     A man with a hard hat is dancing.   \n",
       "1      A young child is riding a horse.   \n",
       "2  A man is feeding a mouse to a snake.   \n",
       "3        A woman is playing the guitar.   \n",
       "4         A woman is playing the flute.   \n",
       "\n",
       "                                     sent_2   sim  \n",
       "0      A man wearing a hard hat is dancing.  5.00  \n",
       "1                A child is riding a horse.  4.75  \n",
       "2  The man is feeding a mouse to the snake.  5.00  \n",
       "3                  A man is playing guitar.  2.40  \n",
       "4                 A man is playing a flute.  2.75  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sts_dev.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Build the Evaluation Graph</h3>\n",
    "\n",
    "Next, we need to build the evaluation graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "sts_input1 = tf.placeholder(tf.string, shape=(None))\n",
    "sts_input2 = tf.placeholder(tf.string, shape=(None))\n",
    "\n",
    "# For evaluation we use exactly normalized rather than\n",
    "# approximately normalized.\n",
    "sts_encode1 = tf.nn.l2_normalize(embed(sts_input1), axis=1)\n",
    "sts_encode2 = tf.nn.l2_normalize(embed(sts_input2), axis=1)\n",
    "cosine_similarities = tf.reduce_sum(tf.multiply(sts_encode1, sts_encode2), axis=1)\n",
    "clip_cosine_similarities = tf.clip_by_value(cosine_similarities, -1.0, 1.0)\n",
    "sim_scores = 1.0 - tf.acos(clip_cosine_similarities)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Evaluate Sentence Embeddings</h3>\n",
    "\n",
    "Finally, we need to create a session and run our evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sts_data = sts_dev #@param [\"sts_dev\", \"sts_test\"] {type:\"raw\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pearson correlation coefficient = 0.5184415561190069\n",
      "p-value = 5.886987186745979e-104\n"
     ]
    }
   ],
   "source": [
    "text_a = sts_data['sent_1'].tolist()\n",
    "text_b = sts_data['sent_2'].tolist()\n",
    "dev_scores = sts_data['sim'].tolist()\n",
    "\n",
    "def run_sts_benchmark(session):\n",
    "  \"\"\"Returns the similarity scores\"\"\"\n",
    "  emba, embb, scores = session.run(\n",
    "      [sts_encode1, sts_encode2, sim_scores],\n",
    "      feed_dict={\n",
    "          sts_input1: text_a,\n",
    "          sts_input2: text_b\n",
    "      })\n",
    "  return scores\n",
    "\n",
    "\n",
    "with tf.Session() as session:\n",
    "  session.run(tf.global_variables_initializer())\n",
    "  session.run(tf.tables_initializer())\n",
    "  scores = run_sts_benchmark(session)\n",
    "\n",
    "pearson_correlation = scipy.stats.pearsonr(scores, dev_scores)\n",
    "print('Pearson correlation coefficient = {0}\\np-value = {1}'.format(\n",
    "    pearson_correlation[0], pearson_correlation[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Extra Credit</h3>\n",
    "\n",
    "For extra credit, re-run this analysis with a different Hub module. Are the results different? If so, how?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Further Reading</h2>\n",
    "\n",
    "We published a [blog post](https://developers.googleblog.com/2018/04/text-embedding-models-contain-bias.html) on how bias can affect text embeddings. It's worth a read!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
